When Noise Helps or Hurts Conditional Density Estimation Near Sparse Data Boundaries

A conditional density estimator predicts the whole distribution of an outcome given inputs, not just its average. Neural versions of these models, such as mixture density networks (a neural network that outputs the weights, means and widths of a Gaussian mixture), overfit badly on small data. A standard fix is to add Gaussian noise to the training pairs. Current practice treats this noise as one isotropic knob, the same size for inputs and outputs and the same size everywhere. We argue that this hides two mechanisms with opposite effects. Noise on the outputs blurs each predicted distribution. Noise on the inputs mixes the predicted distributions at neighbouring inputs. If the input noise is too large, the model forgets that the output depends on the input and falls back to predicting the marginal distribution.

The physics of light in disordered multimode fibers has the same split and the same reversal. Weak random coupling between modes speeds up the formation of a clean, condensed beam, while strong coupling forces all modes to be equally populated and suppresses the condensate. We propose to bring two things across from that physics: the decomposition of a random perturbation into a "within-mode" part and a "between-mode" part, and the competition between two length scales. We then use the WKB method (an asymptotic technique for wave equations whose coefficients vary slowly) to predict where input noise switches from helpful to harmful. Our claim is that this switch happens first and most sharply where input data are sparse, near the edges of the data support, at a noise scale set by how fast the input density vanishes there.

Core Claim

Write the noise-regularized training objective with separate input and output noise scales $h_x$ and $h_y$. Rothfuss et al. showed that noise injection is asymptotically equivalent to a penalty on the Laplacian of the negative log-density (Noise Regularization for Conditional Density Estimation). Splitting their penalty by coordinate shows that the input part acts on the predicted conditional $q(y\mid x)$ like a short heat flow in $x$. Weighted by the input density $p(x)$, that flow is generated by the operator $\mathcal{L}=p^{-1}\nabla\cdot(p\nabla)$. Substituting $\psi=\sqrt{p}\,\phi$ turns its eigenproblem into a Schrödinger equation:

$$-\Delta\psi + V\psi = \mu\psi,\qquad V=\frac{\Delta\sqrt{p}}{\sqrt{p}}.$$

Here $\mu$ is the eigenvalue, and each eigenfunction is one pattern of how the conditional varies with $x$. Input noise damps pattern $k$ by roughly $e^{-h_x^2\mu_k/2}$.

Near a boundary where $p(x)\propto x^{\chi}$ ($x$ is distance to the boundary, $\chi$ the vanishing exponent), the potential becomes inverse-square, $V\approx \chi(\chi-2)/(4x^2)$. This is the regime where WKB needs a boundary correction and where Tomasini, Sclocchi and Wyart found that averaging over the training sample stops being reliable. Their setting was Laplace kernel ridge regression, where they predicted a crossover ridge $\lambda^*\sim P^{-1/(d+\chi)}$ (Failure and success of the spectral bias prediction for Laplace Kernel Ridge Regression). The same counting argument can be applied here. Require that a noise ball of radius $h_x$ at the boundary contains order one sample, so that $n\,h_x^{d}\,h_x^{\chi}\sim 1$, where $n$ is the sample size and $d$ the intrinsic dimension. This gives the crossover

$$h_x^*\sim n^{-1/(d+\chi)}.$$

Below $h_x^*$, the boundary conditionals are driven by individual training points and collapse into spikes. Above it, the low-$\mu$ patterns survive while the localized boundary patterns are washed out. The output noise $h_y$ plays the within-mode role and only competes with the separation between modes in $y$. The specific new content is therefore threefold: input and output noise should be tuned separately; the input-noise optimum should depend on position and grow toward sparse boundaries; and the global crossover should scale with $n$ and $\chi$ as above.

Research Program

Theory. We first derive the decomposition. A second-order expansion of the noisy objective in $x$ and $y$ separately gives two terms. One is a within-conditional term that matches the diagonal part of the disorder matrix in the multimode-fiber kinetic theory (Interplay of thermalization and strong disorder). The other is a cross-$x$ coupling term that matches the off-diagonal part, the part that drives equal population of modes. The physics supplies the organizing picture, not a proof: in fibers, weak disorder gave condensation "within just a few meters" instead of kilometers (Dramatic Acceleration of Wave Condensation Mediated by Disorder in Multimode Fibers). Strong disorder (energy shift $\Delta E/N\approx 19\%$) almost entirely inhibited thermalization (Interplay of thermalization and strong disorder: Wave turbulence theory, numerical simulations, and experiments in multimode optical fibers Interplay of Thermalization and Strong Disorder: Wave Turbulence Theory, Numerical Simulations, and Experiments in Multimode Optical Fibers).

We then solve the weighted-Laplacian spectrum by WKB in one dimension and in slab geometries with $p\propto x^\chi$. This follows the static-operator approach of Tomasini et al., but applies it to the smoothing operator of a density model rather than a regression kernel. From the spectrum we derive the damping of each pattern and the crossover $h_x^*$. Like theirs, our analysis is asymptotic and limited to simple geometries; we state this rather than claim general manifolds.

Method. We build an anisotropic, position-dependent noise scheme. The output noise $h_y$ is a single constant. The input noise $h_x(x)$ follows the predicted profile, using the local nearest-neighbour radius as a density proxy and a boundary exponent $\chi$ estimated from the data.

Architectures. The scheme is applied on top of mixture density networks with softmax gating and with sigmoid gating. The universal-approximation guarantees for softmax mixtures of experts (Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models) tell us that any loss of $x$-dependence comes from regularization, not from lack of model capacity.

Exploratory test of the acceleration analogy. We also test, as a clearly secondary question, whether small input noise speeds up parameter convergence in over-specified softmax mixtures. Those mixtures stall at slow rates because their parameters satisfy a degenerate system of differential equations (Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?). Large noise should reverse any such benefit.

Baselines

Do nothing. The naive reference is an unregularized mixture density network with weight decay and early stopping, using the same architecture and training budget. Rothfuss et al. report that such models overfit into spikes on small data such as Boston Housing (405 training points). On Lacuna there is no reported numerical log-likelihood for this comparison.

Strongest direct competitor. This is isotropic noise regularization as in Noise Regularization for Conditional Density Estimation. It is the established fix and uses exactly the mechanism we refine. We give its single noise scale the same number of cross-validation trials as our method gets in total, on identical splits and seeds. No reported result was found on Lacuna for its test negative log-likelihood on Boston Housing, NYC Taxi or EuroStoxx, so we rerun it ourselves. Beating it means a lower test negative log-likelihood averaged over datasets, with the gain concentrated in the lowest-density decile of test inputs, as the theory predicts.

Strongest non-neural competitor on tabular data. This is CDTree (Conditional Density Estimation with Histogram Trees). It reached the best average rank among interpretable models and the lowest negative log-likelihood on 6 of 14 UCI datasets; per-dataset values are not reported on Lacuna. We run it on the same 14 datasets with its own description-length model selection and report average rank.

Ablations isolating the key ingredient. The first ablation tunes $h_x$ and $h_y$ as separate global constants by cross-validation. It tests whether splitting the noise matters before position dependence is added. The second ablation uses position-dependent input noise with the classical exponent $h_x(x)\propto (n\,p(x))^{-1/d}$ and no boundary term. It tests whether the WKB-derived boundary behaviour adds anything beyond ordinary adaptive bandwidths.

Why not just…?

Why not just cross-validate two global noise scales? This is a real threat on datasets with little boundary mass. The theory predicts that a single global $h_x$ must compromise between interior and boundary. The ablation above measures how much that compromise costs. If the two-constant version matches ours everywhere, including the low-density decile, the position-dependent part of the claim fails. What would remain is the decomposition and the scaling law, which still tell practitioners where to search.

Why not just use classical adaptive bandwidths from kernel density estimation? Variable bandwidths that grow where data are sparse are old and need no WKB. The difference we predict is at boundaries with $\chi\neq 0$. There, the inverse-square potential localizes the conditional's variation patterns, so the right noise depends on distance to the edge and not only on local density. The second ablation settles this directly. If it ties our method, we will report that the WKB analysis explains rather than improves adaptive bandwidths.

Why not just fix the gating, for example by switching to sigmoid gates? Sigmoid gating fixes a different problem: parameter estimation rates. Its reported convergence exponent was $n^{-0.51}$ versus $n^{-0.24}$ for softmax with ReLU experts (Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts). It says nothing about spikes or about losing $x$-dependence on small data. We cross every noise scheme with both gate types. Our claim survives only if the noise effects persist under sigmoid gating.

Proposed Experiments

Decisive synthetic test. We use inputs with density $p(x)\propto x^{\chi}$ on $[0,1]^d$ for $\chi\in\{0,0.5,1.5,3\}$, $d\in\{1,2,4\}$ and $n$ from $10^3$ to $10^5$. The target is a known conditional that is bimodal, with modes splitting as $x$ approaches the boundary. For each setting we sweep $h_x$ and locate the optimum by exact KL divergence. The prediction is that the optimal $h_x$ scales as $n^{-1/(d+\chi)}$, so the fitted exponent should track $1/(d+\chi)$. A fitted exponent that is flat in $\chi$ falsifies the WKB account.

Quench check. We track the model's mutual information between $x$ and $y$ as $h_x$ grows. The theory predicts it collapses toward zero first in the boundary slab, mirroring how strong disorder suppressed condensation in fibers.

Real data. We use Boston Housing, NYC Taxi and EuroStoxx (from Rothfuss et al.) and the 14 UCI datasets (from CDTree). Metrics are test negative log-likelihood overall and in the lowest-density decile, with $\chi$ fitted from learning curves in the manner of Tomasini et al.

Exploratory acceleration test. We reuse the over-specified Gaussian mixture-of-experts setup with $n=10^4$ to $10^5$ from Nguyen et al. and ask whether small $h_x$ improves the empirical parameter-error exponent over the noise-free softmax gate. A null result here does not affect the main claim.