Risk Bounds on Aleatoric Uncertainty Recovery

Yikai Zhang, Jiahe Lin, Fengpei Li, Yeshaya Adler, Kashif Rasul, Anderson Schneider, Yuriy NevmyvakaAISTATS 2023 · OpenReview

Abstract

Quantifying aleatoric uncertainty is a challenging yet essential task in machine learning, particularly for decision-making in environments with data-dependent noise. While common approaches rely on Gaussian likelihood or moment matching, theoretical guarantees for these methods have been sparse. This work establishes Probably Approximately Correct (PAC) risk bounds for aleatoric uncertainty recovery in heteroscedastic regression. We demonstrate that Negative Log-Likelihood (NLL) and Mean Squared Error (MSE) based methods calibrate different aspects of uncertainty, leading to distinct performance advantages in different noise regimes. Our empirical results confirm these theoretical findings across various synthetic settings.

Introduction

In machine learning, uncertainty is typically categorized into two types: epistemic uncertainty, which arises from a lack of data and can be reduced with more observations, and aleatoric uncertainty, which represents the inherent randomness of the data-generating process. While deep learning has made significant strides in reducing epistemic uncertainty, quantifying aleatoric uncertainty remains a complex problem.

Most practitioners estimate data-dependent variance using one of two training objectives: Gaussian Negative Log-Likelihood (NLL) or Mean Squared Error (MSE) via moment matching. This paper provides the first theoretical framework to compare these methods, offering finite-sample risk bounds that explain why their performance varies across different datasets.

Theoretical Framework and PAC-Learnability

We study the heteroscedastic regression setting where $Y = \mu(X) + \xi(X)$, and the noise $\xi(X)$ has a variance $\sigma^2(X)$ that depends on the input $X$. A key contribution of this research is proving that aleatoric uncertainty is indeed PAC-learnable.

To establish these bounds, we utilize Rademacher complexity and Pseudo-dimension. A significant technical innovation in our proof is the use of the Talagrand contraction lemma, which allows for the joint estimation of the mean function $\mu$ and the variance function $\sigma^2$ on the same dataset. This "decoupling" approach makes the theoretical guarantees applicable to modern deep learning practices where joint training is standard.

Methodology and Experimental Setup

The researchers evaluated the estimators using five synthetic settings (S1 through S5). These settings provide a ground truth variance function, allowing for precise measurement of recovery error.

Visualization of the true mean and various variance functions (S1-S5) used to evaluate the estimators.
  • S1-S3: Feature smooth variance functions such as quadratic and B-splines.
  • S4-S5: Utilize complex, discontinuous variance functions to test the limits of the estimators. S5 specifically incorporates heavy-tailed Student-t noise to assess robustness.

Key Insights: The Ratio vs. Additive Trade-off

The theoretical analysis reveals a fundamental trade-off between NLL and MSE estimators based on the noise regime:

  1. NLL and Ratio Error: The NLL objective effectively provides guarantees on the precision function ($1/\sigma^2$). This results in a ratio error minimization, making NLL exceptionally accurate in low-variance regimes where signals are relatively quiet.
  2. MSE and Additive Error: The MSE objective provides direct guarantees on the variance function ($\sigma^2$). It minimizes an additive error, which gives it a significant advantage in high-variance regimes where the ratio-based penalty of NLL becomes overly sensitive to large noise values.

Empirical Results

Empirical tests strikingly confirm these theoretical predictions. By partitioning test data into quartiles based on the true variance, the authors demonstrated the crossover in performance between the two methods.

Comparison of estimator performance across variance quartiles. NLL (orange) excels in low-variance quartiles, while MSE (blue) shows superior stability in high-variance regimes.

As predicted, NLL is the specialist for low-noise signals, while MSE acts as a more robust generalist for high-noise environments. This trend holds even when the function complexity increases or the noise distribution becomes heavy-tailed.

Conclusion

This work transforms aleatoric uncertainty estimation from an empirical "black art" into a principled science. By providing PAC risk bounds, it offers a mathematical roadmap for selecting the appropriate estimator based on the expected noise level of the application. Future research may focus on hybrid or adaptive loss functions that can automatically transition between NLL and MSE objectives to optimize uncertainty recovery across all regimes.

Summary updated

Abstract

Quantifying aleatoric uncertainty is a challenging task in machine learning. It is important for decision making associated with data-dependent uncertainty in model outcomes. Recently, many empirical studies in modeling aleatoric uncertainty under regression settings primarily rely on either a Gaussian likelihood or moment matching. However, the performance of these methods varies for different datasets whereas discussions on their theoretical guarantees are lacking. In this work, we investigate theoretical aspects of these approaches and establish risk bounds for their estimates. We provide conditions that are sufficient to guarantee the PAC-learnablility of the aleatoric uncertainty. The study suggests that the likelihood and moment matching-based methods enjoy different types of guarantee in their risk bounds, i.e., they calibrate different aspects of the uncertainty and thus exhibit distinct properties in different regimes of the parameter space. Finally, we conduct empirical study which shows promising results and supports our theorems.
BibTeX
@inproceedings{zhang2023risk,
  author = {Yikai Zhang and Jiahe Lin and Fengpei Li and Yeshaya Adler and Kashif Rasul and Anderson Schneider and Yuriy Nevmyvaka},
  title = {{Risk Bounds on Aleatoric Uncertainty Recovery}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=wzGoGNXVAa},
}
Figure