TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
TokenGate reports 1.4–11.3× faster time to first token with no accuracy loss across the tasks tested.
The drop-in component learns which input tokens to prune during prefill, reducing peak memory use by 1.5–3.4×. Its token-gated attention mechanism uses importance scores to reweight attention logits, enabling end-to-end training despite discrete token selection. Against the best-performing prior token-pruning methods, the authors report up to 6.4× faster prefill and 2× lower peak memory. It supports standard transformers and hybrid-attention models through lightweight adaptation comparable to parameter-efficient fine-tuning.
HF Daily Papers' note
The drop-in component learns which input tokens to prune during prefill, reducing peak memory use by 1.5–3.4×. Its token-gated attention mechanism uses importance scores to reweight attention logits, enabling end-to-end training despite discrete token selection. Against the best-performing prior token-pruning methods, the authors report up to 6.4× faster prefill and 2× lower peak memory. It supports standard transformers and hybrid-attention models through lightweight adaptation comparable to parameter-efficient fine-tuning.
HF Daily Papers' note
score 5