SEIS: Self-Evolving Inference Systems
SEIS reports a 3.27x throughput gain by letting an agent iteratively rewrite a mini inference engine end to end.
The paper tests the approach on Qwen3-0.6B served on H100, comparing the evolved mini-sglang engine against the original and several production-grade systems. It says the optimized engine beats vLLM, TensorRT-LLM, and SGLang on a single-request workload. The authors also check numerical differences and downstream accuracy on math and long-context retrieval tasks. They argue the gains come from system-wide redesign across inherited optimization sessions, not isolated kernel or memory tweaks. ArXiv · AI/CL/LG's note
The paper tests the approach on Qwen3-0.6B served on H100, comparing the evolved mini-sglang engine against the original and several production-grade systems. It says the optimized engine beats vLLM, TensorRT-LLM, and SGLang on a single-request workload. The authors also check numerical differences and downstream accuracy on math and long-context retrieval tasks. They argue the gains come from system-wide redesign across inherited optimization sessions, not isolated kernel or memory tweaks. ArXiv · AI/CL/LG's note
score 6