Evolving Infra: Agent-Adaptive LLM Serving at Scale
The authors report up to 4.1× lower mean request latency and 14× lower P95 time to first token than NVIDIA’s Dynamo.
Evolving Infra separates compute allocation from KV cache management so nodes can switch efficiently between prefill and decode. An online planner adjusts that split to the workload, while an agent-aware cache manager uses context-management signals to guide eviction and transfer. The evaluations covered coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs.
HF Daily Papers' note
Evolving Infra separates compute allocation from KV cache management so nodes can switch efficiently between prefill and decode. An online planner adjusts that split to the workload, while an agent-aware cache manager uses context-management signals to guide eviction and transfer. The evaluations covered coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs.
HF Daily Papers' note
score 5