VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
VelServe leaves room for slow decoding steps so more requests meet token-delivery deadlines.
The paper finds that a single late decoding step delays every request in its batch, making even rare overruns costly for long responses. VelServe uses measured latency tails to limit prompt-processing work per step, then prioritizes shorter waiting prompts to offset the slowdown. The authors report 2.4–2.9× the strongest baseline’s throughput of requests meeting latency targets on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4× on DeepSeek-V2-Lite. With prompt processing and decoding split across eight GPUs, it meets those targets for more than twice as many requests as the baselines.
ArXiv · AI/CL/LG's note
The paper finds that a single late decoding step delays every request in its batch, making even rare overruns costly for long responses. VelServe uses measured latency tails to limit prompt-processing work per step, then prioritizes shorter waiting prompts to offset the slowdown. The authors report 2.4–2.9× the strongest baseline’s throughput of requests meeting latency targets on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4× on DeepSeek-V2-Lite. With prompt processing and decoding split across eight GPUs, it meets those targets for more than twice as many requests as the baselines.
ArXiv · AI/CL/LG's note
score 5