RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill
RidgeFill reports up to 1.60× faster time to first token, with measured quality tradeoffs.
It skips most upper-layer computation for the prompt, filling their caches using ridge-regression maps fitted once on calibration text. Selected prompt positions, the final few thousand tokens and every decoding step still run through the full model; no tokens are dropped. On Qwen3.8-Flash-Next in vLLM, it achieved 1.50–1.60× speedups at 9K–32K tokens, with needle-retrieval scores of 99.8 versus 100.0 and novel-token perplexity 2.65% higher. Separate 27B tests showed smaller speedups and benchmark declines whose confidence intervals included zero but could not rule out losses of a few points.
HF Daily Papers' note
It skips most upper-layer computation for the prompt, filling their caches using ridge-regression maps fitted once on calibration text. Selected prompt positions, the final few thousand tokens and every decoding step still run through the full model; no tokens are dropped. On Qwen3.8-Flash-Next in vLLM, it achieved 1.50–1.60× speedups at 9K–32K tokens, with needle-retrieval scores of 99.8 versus 100.0 and novel-token perplexity 2.65% higher. Separate 27B tests showed smaller speedups and benchmark declines whose confidence intervals included zero but could not rule out losses of a few points.
HF Daily Papers' note
score 5