RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill
RidgeFill reports up to 1.60× faster time to first token, with measured accuracy costs.
It skips most upper-layer prompt computation, filling their caches from predictions made by ridge-regression maps fitted once on calibration text. Selected prompt rows, the final few thousand tokens and every decoding step still run through the full model; no tokens are dropped. On Qwen3.8-Flash-Next in vLLM, the reported speedup was 1.50–1.60× at 9K–32K tokens, with needle-retrieval scores of 99.8 versus 100.0 and novel-token perplexity 2.65% higher. Separate Qwen3.8-27B tests showed smaller speedups and benchmark declines whose confidence intervals included zero but did not rule out losses of a few points.
ArXiv · AI/CL/LG's note
It skips most upper-layer prompt computation, filling their caches from predictions made by ridge-regression maps fitted once on calibration text. Selected prompt rows, the final few thousand tokens and every decoding step still run through the full model; no tokens are dropped. On Qwen3.8-Flash-Next in vLLM, the reported speedup was 1.50–1.60× at 9K–32K tokens, with needle-retrieval scores of 99.8 versus 100.0 and novel-token perplexity 2.65% higher. Separate Qwen3.8-27B tests showed smaller speedups and benchmark declines whose confidence intervals included zero but did not rule out losses of a few points.
ArXiv · AI/CL/LG's note
score 5