On numerical issues of divergences between nearly equal distributions
Rounding and cancellation can corrupt both divergence values and gradients when categorical distributions are nearly equal.
Brandon Amos identifies the issues in forward and reverse KL divergence, generalized Jensen–Shannon divergence, Monte Carlo KL estimates, and entropy. The errors can affect optimization objectives used in knowledge distillation, reinforcement learning, and proximal methods. The paper proposes numerically stable reformulations and supplies example implementations validated with property-based tests.
ArXiv · AI/CL/LG's note
Brandon Amos identifies the issues in forward and reverse KL divergence, generalized Jensen–Shannon divergence, Monte Carlo KL estimates, and entropy. The errors can affect optimization objectives used in knowledge distillation, reinforcement learning, and proximal methods. The paper proposes numerically stable reformulations and supplies example implementations validated with property-based tests.
ArXiv · AI/CL/LG's note
score 5