Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
External KV caching only pays off past the break-even point for the specific hardware and workload.
The paper benchmarks vLLM KV caching across GPU, CPU, and NVMe tiers and argues that bandwidth alone does not predict performance. Its py-kvcache connector uses async direct I/O, bounded staging memory, and scheduler-aware preloading to overlap disk reads with queued requests. At 80k tokens, disk loading is reported as 2.0x faster than LMCache, though trace replays show H100 runs can fall below the point where external caching helps. ArXiv · AI/CL/LG's note
The paper benchmarks vLLM KV caching across GPU, CPU, and NVMe tiers and argues that bandwidth alone does not predict performance. Its py-kvcache connector uses async direct I/O, bounded staging memory, and scheduler-aware preloading to overlap disk reads with queued requests. At 80k tokens, disk loading is reported as 2.0x faster than LMCache, though trace replays show H100 runs can fall below the point where external caching helps. ArXiv · AI/CL/LG's note
score 6