Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Windowed-MTP keeps speculative decoding cheap at million-token context by limiting only the draft head’s attention window.
The paper says native MTP draft heads can become the bottleneck because they reread the full KV cache at each draft step. Its proposed Windowed-MTP uses a sliding window plus attention sink for draft attention, while the target model still verifies with full attention. At 1M context, the method drops about 99% of draft KV entries and cuts per-decode-step cost by 28% to 44% across the tested Qwen and Mamba2-hybrid systems. The author frames it as training-free and output-distribution preserving because windowing changes proposals, not verified accepted tokens. ArXiv · AI/CL/LG's note
The paper says native MTP draft heads can become the bottleneck because they reread the full KV cache at each draft step. Its proposed Windowed-MTP uses a sliding window plus attention sink for draft attention, while the target model still verifies with full attention. At 1M context, the method drops about 99% of draft KV entries and cuts per-decode-step cost by 28% to 44% across the tested Qwen and Mamba2-hybrid systems. The author frames it as training-free and output-distribution preserving because windowing changes proposals, not verified accepted tokens. ArXiv · AI/CL/LG's note
score 5