BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
BeaconKV uses compact “beacon queries” to predict which old KV pairs a reasoning model will need again.
The paper argues that long chain-of-thought inference breaks a common cache-compression assumption: recent queries do not reliably predict future attention. It identifies “Thought Revisiting Tokens” that return to distant earlier context, and says their queries cluster into a small number of similarity groups. BeaconKV keeps representatives of those groups instead of the full query history. Across four open-source reasoning models, the authors report up to 5.8x memory reduction, near-full-cache accuracy, and more than 4.3x throughput improvement. HF Daily Papers' note
The paper argues that long chain-of-thought inference breaks a common cache-compression assumption: recent queries do not reliably predict future attention. It identifies “Thought Revisiting Tokens” that return to distant earlier context, and says their queries cluster into a small number of similarity groups. BeaconKV keeps representatives of those groups instead of the full query history. Across four open-source reasoning models, the authors report up to 5.8x memory reduction, near-full-cache accuracy, and more than 4.3x throughput improvement. HF Daily Papers' note
score 5