UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
The paper reports stable FP4 pretraining of an 8B Nemotron-H model over nearly 190 billion tokens using UE5M3 block scaling.
The authors pair E2M1 FP4 payloads with unsigned E5M3 block scales, allowing less frequent tensor scaling than NVIDIA’s Transformer Engine recipe. Their setup drops the randomized Hadamard transform, uses selective stochastic rounding for backward gradients, and keeps FP4 in eligible internal linear layers. Against Transformer Engine NVFP4, the block-16 recipe ends with lower training loss and lower held-out validation negative log-likelihood under each method’s quantized-inference policy. An ablation removing RHT and the BF16 final-block exemption raises measured model-body token throughput by 21.2%. ArXiv · AI/CL/LG's note
The authors pair E2M1 FP4 payloads with unsigned E5M3 block scales, allowing less frequent tensor scaling than NVIDIA’s Transformer Engine recipe. Their setup drops the randomized Hadamard transform, uses selective stochastic rounding for backward gradients, and keeps FP4 in eligible internal linear layers. Against Transformer Engine NVFP4, the block-16 recipe ends with lower training loss and lower held-out validation negative log-likelihood under each method’s quantized-inference policy. An ablation removing RHT and the BF16 final-block exemption raises measured model-body token throughput by 21.2%. ArXiv · AI/CL/LG's note
score 6