Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom cuts the host-memory scan cost by letting each query decide how many key bits it needs.
The paper targets long agentic sessions where KV caches and ranking indexes sit off-GPU, making key scans the decode bottleneck. Fathom stores a 4-bit key cache as bit planes and allocates each query’s read budget across channels by importance. On Qwen3-8B at one million tokens, it reports a 1.67x GPU-time speedup over 136-bit scan baselines, while matching exact top-k decoding on RULER-style tasks. It also notes the method is not faster when the index already lives in GPU memory. HF Daily Papers' note
The paper targets long agentic sessions where KV caches and ranking indexes sit off-GPU, making key scans the decode bottleneck. Fathom stores a 4-bit key cache as bit planes and allocates each query’s read budget across channels by importance. On Qwen3-8B at one million tokens, it reports a 1.67x GPU-time speedup over 136-bit scan baselines, while matching exact top-k decoding on RULER-style tasks. It also notes the method is not faster when the index already lives in GPU memory. HF Daily Papers' note
score 5