Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
The paper says tool progress reports beat pre-call duration guesses for deciding what to do with an agent’s KV cache.
The authors argue that agent requests often wait on tools while GPU memory remains tied up. They propose having running tools expose progress signals, either work remaining or a reliable near-finish indicator. In their measurements, those signals were several times to an order of magnitude more accurate than published predictors at KV-cache decision points. Integrated into a production engine, the approach reduced p90 post-tool TTFT by about 20.7% to 20.8% against LRU. ArXiv · AI/CL/LG's note
The authors argue that agent requests often wait on tools while GPU memory remains tied up. They propose having running tools expose progress signals, either work remaining or a reliable near-finish indicator. In their measurements, those signals were several times to an order of magnitude more accurate than published predictors at KV-cache decision points. Integrated into a production engine, the approach reduced p90 post-tool TTFT by about 20.7% to 20.8% against LRU. ArXiv · AI/CL/LG's note
score 5