What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
At matched validation loss, NVFP4 needed about 20% more exposures to learn rare facts under the study’s primary measure.
The researchers trained transformers from scratch using emulated 4-bit matrix multiplications on synthetic biographies, both alone and mixed into web text. NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall, despite validation-loss differences of just 0.5% and 2%. Evaluating both models with BF16 matrix multiplications reduced but did not eliminate the gap. Raising the learning rate failed to recover the exposure cost, and the authors recommend measuring rare-fact acquisition alongside loss.
ArXiv · AI/CL/LG's note
The researchers trained transformers from scratch using emulated 4-bit matrix multiplications on synthetic biographies, both alone and mixed into web text. NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall, despite validation-loss differences of just 0.5% and 2%. Evaluating both models with BF16 matrix multiplications reduced but did not eliminate the gap. Raising the learning rate failed to recover the exposure cost, and the authors recommend measuring rare-fact acquisition alongside loss.
ArXiv · AI/CL/LG's note
score 5