ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
ResKV keeps exact cache entries while adding a residual cache for the attention mass lost when tokens are evicted.
The paper frames omitted tokens as residual statistics in both the numerator and denominator of softmax attention. Its method splits a fixed KV budget between a main cache and compact residual entries that join the same softmax normalization. Allocation is chosen per layer and KV head during construction, with a decode-time gate adjusting residual contributions per query. Evaluations on LongBench and RULER report improvements over compression baselines under the same KV budget while preserving compressed-decoding efficiency. ArXiv · AI/CL/LG's note
The paper frames omitted tokens as residual statistics in both the numerator and denominator of softmax attention. Its method splits a fixed KV budget between a main cache and compact residual entries that join the same softmax normalization. Allocation is chosen per layer and KV head during construction, with a decode-time gate adjusting residual contributions per query. Evaluations on LongBench and RULER report improvements over compression baselines under the same KV budget while preserving compressed-decoding efficiency. ArXiv · AI/CL/LG's note
score 5