Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
The paper cuts the Transformer decode cache by storing grouped values and reconstructing keys only when needed.
GVA uses a learned linear map so content keys do not have to be persistently cached in the intended inference path. A small shared RoPE channel keeps positional information through a separately cached positional key. In the tested setups, the authors report about 45–47% fewer persistent cache scalars than matched GQA, with near-GQA accuracy at 350M parameters. Custom decoding kernels are built but end-to-end inference speed results are still being evaluated. HF Daily Papers' note
GVA uses a learned linear map so content keys do not have to be persistently cached in the intended inference path. A small shared RoPE channel keeps positional information through a separately cached positional key. In the tested setups, the authors report about 45–47% fewer persistent cache scalars than matched GQA, with near-GQA accuracy at 350M parameters. Custom decoding kernels are built but end-to-end inference speed results are still being evaluated. HF Daily Papers' note
score 5