Megadose The AI curve, compressed.

On numerical issues of divergences between nearly equal distributions

· ArXiv · AI/CL/LG ·
Rounding and cancellation can corrupt both divergence values and gradients when categorical distributions are nearly equal.

Brandon Amos identifies the issues in forward and reverse KL divergence, generalized Jensen–Shannon divergence, Monte Carlo KL estimates, and entropy. The errors can affect optimization objectives used in knowledge distillation, reinforcement learning, and proximal methods. The paper proposes numerically stable reformulations and supplies example implementations validated with property-based tests.

ArXiv · AI/CL/LG's note

score 5

Categories: Research