OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
OasisKV uses speculative lookahead to keep only likely-needed KV blocks in GPU HBM during decode.
The paper says decode-time attention is sparse enough that full KV cache storage does not have to stay in HBM. OasisKV predicts future important tokens from speculative decoding drafts, prefetches their KV blocks from host or remote memory, and stages them before the next decode step. Built on vLLM, it reports accuracy within 0.7 points of full attention under a 2,048-token KV budget, with throughput gains up to 2.1x in the tested long-context settings. HF Daily Papers' note
The paper says decode-time attention is sparse enough that full KV cache storage does not have to stay in HBM. OasisKV predicts future important tokens from speculative decoding drafts, prefetches their KV blocks from host or remote memory, and stages them before the next decode step. Built on vLLM, it reports accuracy within 0.7 points of full attention under a 2,048-token KV budget, with throughput gains up to 2.1x in the tested long-context settings. HF Daily Papers' note
score 6