OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
OUTLETS uses speculative decoding’s own draft representations to predict how long an LLM response will run.
The paper argues those draft-decoder signals already carry information useful for length prediction, avoiding a separate proxy model. Its method adds a lightweight regression head when speculative decoding features are already being computed. In the reported serving setup, those predictions helped schedulers cut short-request P99 latency by 34.8%. Accepted to EMNLP 2026. ArXiv · AI/CL/LG's note
The paper argues those draft-decoder signals already carry information useful for length prediction, avoiding a separate proxy model. Its method adds a lightweight regression head when speculative decoding features are already being computed. In the reported serving setup, those predictions helped schedulers cut short-request P99 latency by 34.8%. Accepted to EMNLP 2026. ArXiv · AI/CL/LG's note
score 4