In probabilistic modeling, some rules are absolute. A probability density function (PDF) or a point process intensity function must never drop below zero. If a model predicts a negative probability or a negative event rate, it violates the fundamental logic of the problem domain.
Yet, our most powerful function approximators—such as deep neural networks, Gaussian Processes (GPs), and orthogonal series expansions—are inherently unconstrained. They naturally map inputs to the entire real line ($\mathbb{R}$). When forced to fit complex, noisy data, these flexible models often produce negative values, particularly in data-sparse regions or near sharp boundaries.
To fix this, researchers have historically relied on post-hoc adjustments like clipping negative values to zero or truncating series expansions. However, as argued by Goh et al. in Estimating Orientation Distribution Functions with Probability Density Constraints and Spatial Regularity, post-hoc clipping fails to preserve the mathematical integrity of the distribution, destroys gradients needed for optimization, and violates normalization constraints. Similarly, Kim et al. note in K2IE: Kernel Method-based Kernel Intensity Estimators for Inhomogeneous Poisson Processes that such adjustments lack alignment with underlying statistical theory.
To build robust models, non-negativity must be enforced by design. Modern machine learning tackles this challenge through three primary lenses: algebraic squaring, geometric transformations, and implicit optimization barriers.
The Squaring Trick: Positivity by Design
The most intuitive way to force a real-valued function $g(x)$ to be non-negative is to square it:
$$f(x) = g(x)^2$$
This simple transformation guarantees that $f(x) \ge 0$ everywhere, regardless of what $g(x)$ outputs.
In event modeling, Ong et al. leverage this property in Exact, Fast and Expressive Poisson Point Processes via Squared Neural Families. By squaring the output of a two-layer neural network, they construct a strictly non-negative intensity function. Crucially, because the underlying network structure is kept simple, the squared function can still be integrated in closed form. This preserves mathematical tractability, allowing for exact and rapid likelihood computations without relying on slow numerical integration.
Squaring also unlocks unexpected structural flexibility in mixture models. Traditional mixture models must restrict their component weights to be positive to prevent the overall density from dipping below zero. However, this restriction limits their ability to model sharp boundaries efficiently. In Subtractive Mixture Models via Squaring: Representation and Learning, Loconte et al. show that if you square a linear combination of components, you can safely allow negative weights:
$$f(x) = \left( \sum_{i} w_i \phi_i(x) \right)^2$$
Because the entire expression is squared, the density is guaranteed to be positive. Yet, the negative weights allow components to subtract probability mass from one another. This "subtractive" capability allows the model to capture complex, multi-modal distributions with significantly fewer parameters than standard additive mixtures.
Isometries and the Square-Root Transform
Squaring can also be interpreted through differential geometry. If we represent a probability density function as $f(x) = \psi(x)^2$, the function $\psi(x)$ is known as the "square-root density." Because $f(x)$ must integrate to 1, the function $\psi(x)$ must satisfy:
$$\int \psi(x)^2 dx = 1$$
This condition means that $\psi(x)$ lives on the surface of an infinite-dimensional unit sphere.
In Nonparametric Fisher Geometry with Application to Density Estimation, Holbrook et al. exploit this isometry to simplify nonparametric density estimation. By placing a Gaussian Process prior on the square-root density $\psi(x)$ and mapping it to this spherical geometry, they create the $\chi^2$-process prior. This framework naturally respects the non-negativity and normalization constraints of probability space without requiring complex, computationally expensive sampling schemes to reject invalid functions.
Implicit Barriers via Fisher Information
Instead of explicitly transforming the model's architecture, can we design optimization landscapes that naturally repel negative values?
Consider the Fisher information regularization term often used in optimal transport and density smoothing:
$$I(f) = \int \frac{|\nabla f(x)|^2}{f(x)} dx$$
Notice the presence of the density function $f(x)$ in the denominator. If the density approaches zero anywhere in its domain, the denominator shrinks, causing the Fisher information to blow up toward infinity.
In Computations of Optimal Transport Distance with Fisher Information Regularization, Li et al. demonstrate that this property acts as an intrinsic mathematical barrier. When minimizing an objective function that includes Fisher information regularization, the optimization path is naturally repelled from zero-crossings. The density is prevented from ever becoming negative because crossing into negative territory would require passing through an infinitely high energy barrier. This eliminates the need for explicit inequality constraints or projection steps, smoothing the optimization landscape and enabling fast convergence using standard numerical solvers.
Functional and Algebraic Constraints
When squaring or exponential transformations are mathematically inconvenient—such as when they break linearity or make integration intractable (Fan et al. in Nonstationary Sparse Spectral Permanental Process)—researchers turn to explicit algebraic constraints.
One approach is to restrict the optimization to a space where non-negativity can be verified mathematically. For example, Henrion et al. show in Mean Squared Error Minimization for Inverse Moment Problems that Semidefinite Programming (SDP) can be used to enforce non-negativity across an entire domain when approximating densities from moments.
Alternatively, the constraint can be baked directly into the choice of the kernel space. In Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions, Kim introduces Inverse M-Kernels, proving that certain kernel classes (like the Laplace kernel) can act as universal approximators strictly within the space of non-negative functions. This prevents artificial zero-crossings or nodal line artifacts without adding optimization overhead.
Key Foundational Work
Skewing Methods for Two-Parameter Locally Parametric Density Estimation (2000) — contributed 3 concept elements to this direction.