Abstract
Quantifying aleatoric uncertainty is a challenging yet essential task in machine learning, particularly for decision-making in environments with data-dependent noise. While common approaches rely on Gaussian likelihood or moment matching, theoretical guarantees for these methods have been sparse. This work establishes Probably Approximately Correct (PAC) risk bounds for aleatoric uncertainty recovery in heteroscedastic regression. We demonstrate that Negative Log-Likelihood (NLL) and Mean Squared Error (MSE) based methods calibrate different aspects of uncertainty, leading to distinct performance advantages in different noise regimes. Our empirical results confirm these theoretical findings across various synthetic settings.
Introduction
In machine learning, uncertainty is typically categorized into two types: epistemic uncertainty, which arises from a lack of data and can be reduced with more observations, and aleatoric uncertainty, which represents the inherent randomness of the data-generating process. While deep learning has made significant strides in reducing epistemic uncertainty, quantifying aleatoric uncertainty remains a complex problem.
Most practitioners estimate data-dependent variance using one of two training objectives: Gaussian Negative Log-Likelihood (NLL) or Mean Squared Error (MSE) via moment matching. This paper provides the first theoretical framework to compare these methods, offering finite-sample risk bounds that explain why their performance varies across different datasets.
Theoretical Framework and PAC-Learnability
We study the heteroscedastic regression setting where $Y = \mu(X) + \xi(X)$, and the noise $\xi(X)$ has a variance $\sigma^2(X)$ that depends on the input $X$. A key contribution of this research is proving that aleatoric uncertainty is indeed PAC-learnable.
To establish these bounds, we utilize Rademacher complexity and Pseudo-dimension. A significant technical innovation in our proof is the use of the Talagrand contraction lemma, which allows for the joint estimation of the mean function $\mu$ and the variance function $\sigma^2$ on the same dataset. This "decoupling" approach makes the theoretical guarantees applicable to modern deep learning practices where joint training is standard.
Methodology and Experimental Setup
The researchers evaluated the estimators using five synthetic settings (S1 through S5). These settings provide a ground truth variance function, allowing for precise measurement of recovery error.
- S1-S3: Feature smooth variance functions such as quadratic and B-splines.
- S4-S5: Utilize complex, discontinuous variance functions to test the limits of the estimators. S5 specifically incorporates heavy-tailed Student-t noise to assess robustness.
Key Insights: The Ratio vs. Additive Trade-off
The theoretical analysis reveals a fundamental trade-off between NLL and MSE estimators based on the noise regime:
- NLL and Ratio Error: The NLL objective effectively provides guarantees on the precision function ($1/\sigma^2$). This results in a ratio error minimization, making NLL exceptionally accurate in low-variance regimes where signals are relatively quiet.
- MSE and Additive Error: The MSE objective provides direct guarantees on the variance function ($\sigma^2$). It minimizes an additive error, which gives it a significant advantage in high-variance regimes where the ratio-based penalty of NLL becomes overly sensitive to large noise values.
Empirical Results
Empirical tests strikingly confirm these theoretical predictions. By partitioning test data into quartiles based on the true variance, the authors demonstrated the crossover in performance between the two methods.
As predicted, NLL is the specialist for low-noise signals, while MSE acts as a more robust generalist for high-noise environments. This trend holds even when the function complexity increases or the noise distribution becomes heavy-tailed.
Conclusion
This work transforms aleatoric uncertainty estimation from an empirical "black art" into a principled science. By providing PAC risk bounds, it offers a mathematical roadmap for selecting the appropriate estimator based on the expected noise level of the application. Future research may focus on hybrid or adaptive loss functions that can automatically transition between NLL and MSE objectives to optimize uncertainty recovery across all regimes.
Summary updated