LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers
LoopFill reports average decoding speedups of 5.85× on Ouro-2.6B and 5.13× on Huginn-3.5B across seven benchmarks.
Looped Transformers reuse the same layers, but repeated passes make token generation slow. The training-free method combines multi-token suffix drafts with shallow-loop predictions in one draft tree, allowing several future tokens to enter the pipeline together. Full-depth verification preserves the model’s autoregressive output distribution. The reported speedups exceed those of LoopAhead, the strongest baseline tested, by 40% and 25% on the two models, respectively.
ArXiv · AI/CL/LG's note
Looped Transformers reuse the same layers, but repeated passes make token generation slow. The training-free method combines multi-token suffix drafts with shallow-loop predictions in one draft tree, allowing several future tokens to enter the pipeline together. Full-depth verification preserves the model’s autoregressive output distribution. The reported speedups exceed those of LoopAhead, the strongest baseline tested, by 40% and 25% on the two models, respectively.
ArXiv · AI/CL/LG's note
score 5