Logarithmic Representations for Numerical Stability in High Dimensions

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2133 papers194 conceptsfrozen 2026-10-04

In high-dimensional machine learning, we often encounter a "numerical cliff." As the number of dimensions increases, probability densities do not just get smaller—they decay exponentially. In a 100-dimensional space, a modest probability in each dimension can easily result in a joint probability of $10^{-100}$, a value far below the "machine epsilon" of standard 32-bit floating-point numbers. When these values hit zero, gradients vanish, training stalls, and models collapse.

To combat this, researchers across generative modeling, reinforcement learning, and Bayesian inference have converged on a shared strategy: shifting the entire computational pipeline into the logarithmic domain. By representing probabilities as log-densities, we transform catastrophic multiplications into stable additions.

The Underflow Problem in High Dimensions

The fundamental issue is the dynamic range of floating-point hardware. Standard IEEE 754 single-precision floats (fp32) can represent numbers as small as roughly $10^{-38}$. In modern deep learning, we frequently model spaces with hundreds or thousands of dimensions. As noted by Filip Rydin in the context of high-dimensional Bayesian filtering, raw density values in these systems often fall below machine precision almost immediately.

When a probability underflows to zero, any subsequent multiplication—common in Bayesian updates or chain rules—remains zero. This "numerical collapse" is particularly lethal for gradient-based optimization. Faguo Wu and colleagues, working on Physics-Informed Normalizing Flows (PINF), argue that parameterizing the logarithm of the density rather than the density itself is the only way to maintain a stable gradient signal when solving high-dimensional Fokker-Planck equations.

From Multiplicative to Additive Updates

The core intuition behind log-space computation is the transformation of the mathematical operations themselves. In the linear domain, combining independent probabilities requires multiplication: $P(A, B) = P(A) \times P(B)$. In the log-domain, this becomes a simple addition: $\log P(A, B) = \log P(A) + \log P(B)$.

This shift is not merely a convenience; it changes the topology of the loss landscape. Steve Renals and Hervé Bourlard demonstrated as early as 1991 that formulating network outputs in the log-likelihood domain prevents the problematic gradients associated with the inverse of small probability values.

More recently, this principle has been applied to recurrent architectures. Leo Feng and Yoshua Bengio showed in their work on minimal RNNs that implementing recurrent updates in log-space prevents the exponential growth or decay of gradients, allowing linear recurrences to scale to much longer sequences without losing precision.

The Log-Sum-Exp Trick

The most significant challenge in log-space is addition. If we have $\log(A)$ and $\log(B)$, how do we find $\log(A+B)$ without ever calculating the raw values of $A$ or $B$? The solution is the Log-Sum-Exp (LSE) trick: $$\log(e^x + e^y) = a + \log(e^{x-a} + e^{y-a})$$ where $a = \max(x, y)$. By factoring out the maximum value, we ensure that the terms inside the exponent are always $\le 0$, preventing numerical overflow.

This trick is the "workhorse" of stable high-dimensional inference. Robert Peharz utilized a log-space variant of tensor contractions in Einsum Networks to scale probabilistic circuits to large image datasets. Similarly, Marko Tuononen applied this principle to Mutual Information estimation, allowing the calculation of normalization factors in hundreds of dimensions by avoiding the direct exponentiation of distances.

Applications Across the Frontier

The move to log-representations is now a standard requirement for scaling complex models:

  • Manifold Learning: In Moser Flow, Noam Rozen and Aditya Grover use log-scale computations to handle the vanishingly small probabilities typical of distributions on complex manifolds like spheres and tori.
  • Reinforcement Learning: Johan Björck found that Soft Actor-Critic can only run in low-precision (fp16) if Gaussian log-probabilities are carefully reorganized to avoid squaring small standard deviations near action boundaries.
  • Rare Event Modeling: For multivariate distributions, Jian Cao and Marc G. Genton developed the tlrmvnmvt package, which estimates log-probabilities to quantify extreme tail risks that would otherwise be rounded to zero by standard machine precision.

By treating the logarithm not just as a transformation, but as the primary domain of computation, these researchers have turned the "Curse of Dimensionality" from a numerical dead-end into a manageable engineering challenge. The reader should take away that in high dimensions, the raw probability is often a ghost—only its logarithm contains the signal necessary for learning.

Papers in this version

Showing 60 of 133 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 159 papers
  2. Version 1 · 2026-10-04 · 133 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_numerical_underflow_f35da895,
  title = {numerical underflow},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-underflow/rdr_f35da8959bc64856b639235e2eab937f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_logarithmic_representations_for_numerica_f35da895_v1,
  title = {Logarithmic Representations for Numerical Stability in High Dimensions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logarithmic-representations-for-numerical-stability-in-high-dimensions/rdr_f35da8959bc64856b639235e2eab937f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}