Standard implementations of machine learning's most common distance measures—like Kullback–Leibler (KL) divergence—have a hidden weakness: they fall apart when two distributions are nearly identical. In a new paper, Brandon Amos from Reflection AI identifies five critical numerical failures in these operations and provides the "fixed" math needed to keep optimization stable.
TL;DR
Standard divergence computations (KL, JSD, Entropy) suffer from catastrophic cancellation and rounding errors when distributions approach equality, leading to noisy or vanishing gradients. This paper introduces numerically stable reformulations using logit-space reconstructions and Taylor expansions, ensuring accuracy in sensitive applications like knowledge distillation and reinforcement learning.
What Problem Does This Paper Address?
In many machine learning workflows, we want one probability distribution ($q$) to match a reference distribution ($p$). This happens in knowledge distillation (student matching a teacher), reinforcement learning (staying close to a reference policy), and proximal optimization.
Mathematically, as $q$ approaches $p$, the divergence between them approaches zero. However, floating-point arithmetic is notoriously bad at handling the difference between two nearly equal, large numbers—a phenomenon called catastrophic cancellation. Standard implementations compute $\log p$ and $\log q$ separately and then subtract them. If the distributions are close enough, the tiny difference between them is rounded away to zero, or worse, becomes pure numerical noise. This results in gradients that either vanish prematurely or push the model in random, incorrect directions.
Methods
The paper proposes eight specific reformulations to bypass these issues. The core strategy involves moving away from "compute then subtract" toward "reconstruct the difference directly."
- Log-ratio Reconstruction: Instead of subtracting separately computed log-probabilities, the authors reconstruct the log-ratio $s = \log q - \log p$ directly from logit differences. This preserves the "signal" that would otherwise be lost during normalization.
- Nonnegative-remainder Identity: For the forward KL computation, the authors replace the standard signed sum with a sum of non-negative terms using the identity $E_2(x) = e^x - 1 - x$.
- Taylor Expansions: To evaluate $E_2(x)$ accurately when $x$ is very near zero, they use a Taylor series expansion. This avoids the $1 - 1$ cancellation that occurs in the standard
exp(x) - 1 - xcalculation. - Zero-sum Gradient Correction: The paper applies a correction to gradients to ensure they satisfy the "zero-sum" property (probabilities must sum to 1), which reduces accumulated roundoff error during backpropagation.
Datasets
The paper does not use standard large-scale training datasets like ImageNet. Instead, it employs property-based testing using the Hypothesis library.
- Evaluation: Implementations are validated against high-precision reference values computed using
mpmathwith 80 decimal digits of precision. - Scope: The tests cover various categorical settings, including different numbers of categories ($K \in \{2, 3, 4, 8, 16\}$) and representable logit shifts.
Key Insights and Ideas
- The Gradient Paradox: A divergence value might look numerically "fine" (e.g., a small positive number), but its gradient can be completely destroyed by rounding. Figure 1 illustrates this: as the perturbation $h$ between distributions decreases, the relative error of the "Basic" implementation's gradient explodes, while the "Fixed" version remains accurate.
- The "Nearly Equal" Guard: The stable reformulations are specialized for the regime where distributions are close. If used when distributions are far apart, the intermediate exponentials can actually cause overflows (see Figure 2). Practical implementations require a "closeness guard" to switch between standard and stable modes.
- Monte Carlo Stability: The paper shows that even Monte Carlo estimators (used when the number of categories is too large to sum over) inherit these rounding issues. Simply regrouping the four scalar inputs used to compute the sampled log-ratio can significantly improve accuracy.
Key Results
The quantitative results are presented as relative error plots comparing "Basic" (standard) vs. "Fixed" (proposed) implementations:
- Forward KL: The basic implementation's relative error for the loss and gradient reaches $10^6$ (or 1,000,000%) as the distributions converge. The fixed version maintains an error near machine epsilon ($\approx 10^{-7}$ for FP32).
- Entropy: The paper demonstrates that while the entropy value might remain accurate near a uniform distribution, the gradient often rounds to zero in standard FP32, a failure fixed by the proposed centering method.
Limitations and Future Work
- Precision Limits: Even with these fixes, if the input logits themselves are already rounded or noisy, the reformulations cannot recover that lost information.
- Lower Precision Formats: The study focused on FP32. The authors note that these issues are significantly exacerbated in lower-precision formats like BF16, FP8, or FP4, which are increasingly common in hardware accelerators.
- Production Integration: The paper provides JAX examples but notes that full production-ready libraries would need robust hybrid logic to handle all possible logit ranges without overflow.
Summary updated