LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
LowRAM reports up to 5× higher peak decoding throughput with a KV cache compressed by 4×.
The method predicts each attention head’s values from its keys, storing residuals only for the tokens it reconstructs worst. It combines that value compression with low-rank key compression across layers, staying close to lossless across three model families. Reconstructing only selected key-value pairs at each decoding step limits decoding overhead. Combined with quantisation and eviction, it reaches 26.9× compression while retaining 94.5% of baseline performance on long contexts.
HF Daily Papers' note
The method predicts each attention head’s values from its keys, storing residuals only for the tokens it reconstructs worst. It combines that value compression with low-rank key compression across layers, staying close to lossless across three model families. Reconstructing only selected key-value pairs at each decoding step limits decoding overhead. Combined with quantisation and eviction, it reaches 26.9× compression while retaining 94.5% of baseline performance on long contexts.
HF Daily Papers' note
score 5