Latent Compaction with Test-Time Training
At 32× cache compression, Latent Compaction scored 96.7% accuracy on RULER, versus 40.0% for KVZip+.
The method trains a compact, document-specific key–value cache while keeping model weights frozen. It first matches attention outputs and normalization terms from the full cache, then jointly optimizes the compact cache through distillation. Training uses generic document prompts without access to downstream questions or answers. The authors report gains over budget-matched baselines on RULER, LongBench and Phonebook Lookup, with retrieval improvements on both Qwen3 and MiniCPM5.
ArXiv · AI/CL/LG's note
The method trains a compact, document-specific key–value cache while keeping model weights frozen. It first matches attention outputs and normalization terms from the full cache, then jointly optimizes the compact cache through distillation. Training uses generic document prompts without access to downstream questions or answers. The authors report gains over budget-matched baselines on RULER, LongBench and Phonebook Lookup, with retrieval improvements on both Qwen3 and MiniCPM5.
ArXiv · AI/CL/LG's note
score 5