Hardware-Aware FP4 FlashAttention-4
FP4 speeds the matrix math, but this paper says attention only gets faster if softmax and quantization are redesigned around the hardware.
Robert Hu introduces Direct-P for noncausal inference, mapping attention scores directly into FP4 probabilities. On an NVIDIA GB200, it reaches up to 2.13x BF16 forward throughput. For causal training, the paper reuses forward quantization in backward and reports up to a 1.14x speedup for a full single-GPU 8B-parameter update. It also says matched distributed training kept FP8 probabilities and values, while every tested MXFP4 probability/value trajectory diverged. ArXiv · AI/CL/LG's note
Robert Hu introduces Direct-P for noncausal inference, mapping attention scores directly into FP4 probabilities. On an NVIDIA GB200, it reaches up to 2.13x BF16 forward throughput. For causal training, the paper reuses forward quantization in backward and reports up to a 1.14x speedup for a full single-GPU 8B-parameter update. It also says matched distributed training kept FP8 probabilities and values, while every tested MXFP4 probability/value trajectory diverged. ArXiv · AI/CL/LG's note
score 6