BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
The authors report 2.4–3.0× higher peak serving throughput than BF16 KV caching on NVIDIA B300.
That result comes from end-to-end tests with Qwen3-4B and Qwen3-30B-A3B. Across the models evaluated, BitDecoding-FP4 stays within 0.5 accuracy points of BF16 KV with NVFP4, and within 1 point with MXFP4. The method addresses query quantization error, probabilities rounding to zero, and key outliers, while fused GPU operations reduce the softmax bottleneck. The authors also report accuracy tracking BF16 KV during reinforcement-learning training rollouts.
ArXiv · AI/CL/LG's note
That result comes from end-to-end tests with Qwen3-4B and Qwen3-30B-A3B. Across the models evaluated, BitDecoding-FP4 stays within 0.5 accuracy points of BF16 KV with NVFP4, and within 1 point with MXFP4. The method addresses query quantization error, probabilities rounding to zero, and key outliers, while fused GPU operations reduce the softmax bottleneck. The authors also report accuracy tracking BF16 KV during reinforcement-learning training rollouts.
ArXiv · AI/CL/LG's note
score 5