InferScale: GPU-Native KV Injection for Personalized LLM Serving
InferScale cuts personalized-memory serving latency by caching reusable KV state on the GPU.
The paper targets LLM systems that repeatedly inject the same retrieved user memories into prompts, raising time-to-first-token as retrieval budgets grow. InferScale precomputes KV representations for memory facts, stores them with semantic embeddings, and injects selected KV directly into vLLM’s paged cache. The authors add Chunked RoPE and Context-Window Encoding to handle dynamic memory assembly and preserve some surrounding context. On LoCoMo across three open-weight models, they report 72–79% lower TTFT at k=50, 60.3% accuracy versus Mem0’s 63.3%, and 3.7–4.5x higher throughput under concurrent load. ArXiv · AI/CL/LG's note
The paper targets LLM systems that repeatedly inject the same retrieved user memories into prompts, raising time-to-first-token as retrieval budgets grow. InferScale precomputes KV representations for memory facts, stores them with semantic embeddings, and injects selected KV directly into vLLM’s paged cache. The authors add Chunked RoPE and Context-Window Encoding to handle dynamic memory assembly and preserve some surrounding context. On LoCoMo across three open-weight models, they report 72–79% lower TTFT at k=50, 60.3% accuracy versus Mem0’s 63.3%, and 3.7–4.5x higher throughput under concurrent load. ArXiv · AI/CL/LG's note
score 5