On numerical issues of divergences between nearly equal distributions

Brandon AmosarXiv 2026 · arXiv

Standard implementations of machine learning's most common distance measures—like Kullback–Leibler (KL) divergence—have a hidden weakness: they fall apart when two distributions are nearly identical. In a new paper, Brandon Amos from Reflection AI identifies five critical numerical failures in these operations and provides the "fixed" math needed to keep optimization stable.

TL;DR

Standard divergence computations (KL, JSD, Entropy) suffer from catastrophic cancellation and rounding errors when distributions approach equality, leading to noisy or vanishing gradients. This paper introduces numerically stable reformulations using logit-space reconstructions and Taylor expansions, ensuring accuracy in sensitive applications like knowledge distillation and reinforcement learning.

What Problem Does This Paper Address?

In many machine learning workflows, we want one probability distribution ($q$) to match a reference distribution ($p$). This happens in knowledge distillation (student matching a teacher), reinforcement learning (staying close to a reference policy), and proximal optimization.

Mathematically, as $q$ approaches $p$, the divergence between them approaches zero. However, floating-point arithmetic is notoriously bad at handling the difference between two nearly equal, large numbers—a phenomenon called catastrophic cancellation. Standard implementations compute $\log p$ and $\log q$ separately and then subtract them. If the distributions are close enough, the tiny difference between them is rounded away to zero, or worse, becomes pure numerical noise. This results in gradients that either vanish prematurely or push the model in random, incorrect directions.

Methods

The paper proposes eight specific reformulations to bypass these issues. The core strategy involves moving away from "compute then subtract" toward "reconstruct the difference directly."

  1. Log-ratio Reconstruction: Instead of subtracting separately computed log-probabilities, the authors reconstruct the log-ratio $s = \log q - \log p$ directly from logit differences. This preserves the "signal" that would otherwise be lost during normalization.
  2. Nonnegative-remainder Identity: For the forward KL computation, the authors replace the standard signed sum with a sum of non-negative terms using the identity $E_2(x) = e^x - 1 - x$.
  3. Taylor Expansions: To evaluate $E_2(x)$ accurately when $x$ is very near zero, they use a Taylor series expansion. This avoids the $1 - 1$ cancellation that occurs in the standard exp(x) - 1 - x calculation.
  4. Zero-sum Gradient Correction: The paper applies a correction to gradients to ensure they satisfy the "zero-sum" property (probabilities must sum to 1), which reduces accumulated roundoff error during backpropagation.

Datasets

The paper does not use standard large-scale training datasets like ImageNet. Instead, it employs property-based testing using the Hypothesis library.

  • Evaluation: Implementations are validated against high-precision reference values computed using mpmath with 80 decimal digits of precision.
  • Scope: The tests cover various categorical settings, including different numbers of categories ($K \in \{2, 3, 4, 8, 16\}$) and representable logit shifts.

Key Insights and Ideas

  • The Gradient Paradox: A divergence value might look numerically "fine" (e.g., a small positive number), but its gradient can be completely destroyed by rounding. Figure 1 illustrates this: as the perturbation $h$ between distributions decreases, the relative error of the "Basic" implementation's gradient explodes, while the "Fixed" version remains accurate.
Rounding and cancellation distort forward KL and its gradient. The "Basic" implementation (blue dashed) shows massive relative error as distributions approach equality, while the "Fixed" version (gray) remains stable.
  • The "Nearly Equal" Guard: The stable reformulations are specialized for the regime where distributions are close. If used when distributions are far apart, the intermediate exponentials can actually cause overflows (see Figure 2). Practical implementations require a "closeness guard" to switch between standard and stable modes.
  • Monte Carlo Stability: The paper shows that even Monte Carlo estimators (used when the number of categories is too large to sum over) inherit these rounding issues. Simply regrouping the four scalar inputs used to compute the sampled log-ratio can significantly improve accuracy.

Key Results

The quantitative results are presented as relative error plots comparing "Basic" (standard) vs. "Fixed" (proposed) implementations:

  • Forward KL: The basic implementation's relative error for the loss and gradient reaches $10^6$ (or 1,000,000%) as the distributions converge. The fixed version maintains an error near machine epsilon ($\approx 10^{-7}$ for FP32).
  • Entropy: The paper demonstrates that while the entropy value might remain accurate near a uniform distribution, the gradient often rounds to zero in standard FP32, a failure fixed by the proposed centering method.
An accurate entropy value can accompany an inaccurate gradient. The "Fixed" version retains the small derivative that standard autodiff rounds to zero.

Limitations and Future Work

  • Precision Limits: Even with these fixes, if the input logits themselves are already rounded or noisy, the reformulations cannot recover that lost information.
  • Lower Precision Formats: The study focused on FP32. The authors note that these issues are significantly exacerbated in lower-precision formats like BF16, FP8, or FP4, which are increasingly common in hardware accelerators.
  • Production Integration: The paper provides JAX examples but notes that full production-ready libraries would need robust hybrid logic to handle all possible logit ranges without overflow.

Summary updated

Abstract

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.
BibTeX
@misc{amos2026numerical,
  author = {Brandon Amos},
  title = {{On numerical issues of divergences between nearly equal distributions}},
  year = {2026},
  eprint = {2611.18511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18511},
}
Figure