TokTier: Exact Stateful Tokenization for Agentic LLM Serving
TokTier targets the tokenizer as a live serving bottleneck, not a preprocessing detail.
The paper says agent workloads often resend long transcripts after small tool outputs, making full re-tokenization expensive even when KV cache hits are high. TokTier keeps tokenization state per session, repairs only the append boundary when possible, and falls back when its stable-boundary check fails. The authors report zero divergence in large differential tests across 17 tokenizer families. In vLLM runs, they report median time-to-first-token drops of 16-34% and much higher throughput under a 50 ms P99 target. ArXiv · AI/CL/LG's note
The paper says agent workloads often resend long transcripts after small tool outputs, making full re-tokenization expensive even when KV cache hits are high. TokTier keeps tokenization state per session, repairs only the append boundary when possible, and falls back when its stable-boundary check fails. The authors report zero divergence in large differential tests across 17 tokenizer families. In vLLM runs, they report median time-to-first-token drops of 16-34% and much higher throughput under a 50 ms P99 target. ArXiv · AI/CL/LG's note
score 5