Intent-Aware Caching for Efficient LLM Serving
The authors report a 34.6% cache hit rate and up to 46.1% lower average serving latency in tests on real conversation workloads.
Intent-Aware Caching uses request intent to help decide which cached conversation states to evict. The authors find that intents differ in how often conversations continue and how quickly users send another turn. Their policy classifies requests and learns reuse statistics from recent serving logs. In evaluations on WildChat and ShareChat, the reported hit rate compares with 18.8% for a previous learning-based approach and 12.2% for least recently used caching.
ArXiv · AI/CL/LG's note
Intent-Aware Caching uses request intent to help decide which cached conversation states to evict. The authors find that intents differ in how often conversations continue and how quickly users send another turn. Their policy classifies requests and learns reuse statistics from recent serving logs. In evaluations on WildChat and ShareChat, the reported hit rate compares with 18.8% for a previous learning-based approach and 12.2% for least recently used caching.
ArXiv · AI/CL/LG's note
score 4