Towards Full Pipeline FP8 Reinforcement Learning for LLMs
FP8 RL training can fail mid-run because quantization noise breaks the importance-ratio clipping logic.
The paper says full-pipeline FP8 reinforcement learning can produce entropy spikes and garbled outputs even when train-inference mismatch is addressed. The authors trace the failure to distorted importance ratios that zero out gradients for too many negative-advantage tokens, letting bad outputs accumulate instead of being penalized. Their proposed fix, Calibrated Clipping, adjusts FP8 clipping bounds to better match BF16 behavior. In tests across GRPO and DAPO, 8B to 32B models, and several FP8 scaling setups, it removes the instability and recovers BF16-level performance. HF Daily Papers' note
The paper says full-pipeline FP8 reinforcement learning can produce entropy spikes and garbled outputs even when train-inference mismatch is addressed. The authors trace the failure to distorted importance ratios that zero out gradients for too many negative-advantage tokens, letting bad outputs accumulate instead of being penalized. Their proposed fix, Calibrated Clipping, adjusts FP8 clipping bounds to better match BF16 behavior. In tests across GRPO and DAPO, 8B to 32B models, and several FP8 scaling setups, it removes the instability and recovers BF16-level performance. HF Daily Papers' note
score 5