Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
The paper finds greedy LLM decoding can change outputs just by switching BF16 to FP16 on the same hardware.
In the authors’ tests, 49–100% of prompts diverged across six models and three benchmarks. They trace many flips to small top-two logit margins at the LM head, where a single token change can cascade into a different output path. Their strongest low-overhead fix is selective FP32 recomputation at the LM head, which improves exact agreement with under 4% latency overhead in low-batch inference. The mitigation is partial: it fails when body-originated error dominates, including batch size 8 or higher and end-to-end FP8 in their tests. ArXiv · AI/CL/LG's note
In the authors’ tests, 49–100% of prompts diverged across six models and three benchmarks. They trace many flips to small top-two logit margins at the LM head, where a single token change can cascade into a different output path. Their strongest low-overhead fix is selective FP32 recomputation at the LM head, which improves exact agreement with under 4% latency overhead in low-batch inference. The mitigation is partial: it fails when body-originated error dominates, including batch size 8 or higher and end-to-end FP8 in their tests. ArXiv · AI/CL/LG's note
score 5