Megadose The AI curve, compressed.

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

· HF Daily Papers ·
The authors report 2.4–3.0× higher peak serving throughput than BF16 KV caching on NVIDIA B300.

Those serving tests used Qwen3-4B and Qwen3-30B-A3B. Across the evaluated dense and mixture-of-experts models, accuracy stayed within 0.5 points of BF16 with NVFP4 and within 1 point with MXFP4. The method tackles query quantization error, probabilities rounding to zero, and key outliers, while fused GPU operations reduce the softmax bottleneck. It also tracked BF16 KV accuracy in reinforcement-learning training rollouts.

HF Daily Papers' note

score 5

Categories: Research