Megadose The AI curve, compressed.

Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation

· ArXiv · AI/CL/LG ·
Callo delivered 1.05–1.19× speedups over vLLM at moderate load in a two-GPU test.

Context edits in agent sessions can invalidate cached KV state, forcing decoding to wait while that state is rebuilt. Callo uses the invalidated state to draft tokens, then checks those drafts against the exact rebuilt state before committing them. It schedules speculation when decode resources are underused and preempts it when drafts are unlikely to be accepted. Tests used SWE-agent sessions across three model families; trace-driven cluster simulations also showed gains at scale.

ArXiv · AI/CL/LG's note

score 5

Categories: Research