On the Fundamental Limits of Prefill-Based Decoding Length Prediction
Randomness in token generation puts a floor under how accurately prefill states can predict response length.
The authors derive error bounds for predictors that use only hidden states computed before generation begins. Under fixed model, dataset and sampling settings, several predictor designs perform similarly close to those bounds, suggesting decoding randomness dominates predictor error. They also find that changing those settings can sharply reduce reported accuracy even when the predictor is unchanged. They recommend holding those conditions fixed in benchmarks and reporting relative errors alongside absolute ones.
ArXiv · AI/CL/LG's note
The authors derive error bounds for predictors that use only hidden states computed before generation begins. Under fixed model, dataset and sampling settings, several predictor designs perform similarly close to those bounds, suggesting decoding randomness dominates predictor error. They also find that changing those settings can sharply reduce reported accuracy even when the predictor is unchanged. They recommend holding those conditions fixed in benchmarks and reporting relative errors alongside absolute ones.
ArXiv · AI/CL/LG's note
score 4