Megadose The AI curve, compressed.

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

· ArXiv · AI/CL/LG ·
VelServe leaves room for slow decoding steps so more requests meet token-delivery deadlines.

The paper finds that a single late decoding step delays every request in its batch, making even rare overruns costly for long responses. VelServe uses measured latency tails to limit prompt-processing work per step, then prioritizes shorter waiting prompts to offset the slowdown. The authors report 2.4–2.9× the strongest baseline’s throughput of requests meeting latency targets on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4× on DeepSeek-V2-Lite. With prompt processing and decoding split across eight GPUs, it meets those targets for more than twice as many requests as the baselines.

ArXiv · AI/CL/LG's note

score 5

Categories: Research