When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Sharding and asynchronous loading cut p99 time-to-first-token from 83.32 seconds to 0.67 seconds on a cache-heavy request stream.
In the GLM-5.3 deployment studied, a remote cache hit initially took 0.88 seconds, versus 0.31 seconds to recompute the prompt. Synchronous loading drove long queues while GPUs were only 7% busy. Partitioning the store across context-parallel ranks increased retrieval bandwidth 6.3-fold; asynchronous loading and fixes for three client bugs completed the changes. The sharded store raised prefill throughput by 14%, and all changes were submitted upstream to LMCache.
HF Daily Papers' note
In the GLM-5.3 deployment studied, a remote cache hit initially took 0.88 seconds, versus 0.31 seconds to recompute the prompt. Synchronous loading drove long queues while GPUs were only 7% busy. Partitioning the store across context-parallel ranks increased retrieval bandwidth 6.3-fold; asynchronous loading and fixes for three client bugs completed the changes. The sharded store raised prefill throughput by 14%, and all changes were submitted upstream to LMCache.
HF Daily Papers' note
score 5