NOMU: Neural Optimization-based Model Uncertainty

Jakob Heiss, Jakob Weissteiner, Hanna Wutte, Sven Seuken, Josef Teichmann

Introduction: The Problem of "Broken" Uncertainty

In high-stakes machine learning applications such as robotics, drug discovery, and Bayesian Optimization (BO), quantifying epistemic uncertainty—knowing what a model does not know—is as critical as the prediction itself. While Gaussian Processes (GPs) provide elegant uncertainty estimates, they often struggle to scale with the complexity of modern neural networks (NNs).

Standard NN uncertainty methods frequently fail to provide logically consistent bounds. For instance, Monte Carlo Dropout (MCDO), a common industry standard, often produces "tubular" uncertainty bounds that do not narrow even at training data points. Similarly, Deep Ensembles (DE) can exhibit zero uncertainty in data gaps where they should be most uncertain. NOMU (Neural Optimization-based Model Uncertainty) addresses these failures by isolating "pure" model uncertainty in noiseless settings to create a method that obeys logical rules of uncertainty.

Comparison of uncertainty bounds: NOMU vs. MC Dropout (MCDO). NOMU correctly narrows at data points, while MCDO remains 'tubular'.

Methods: An Optimization Paradigm

NOMU departs from the traditional Bayesian sampling paradigm, which attempts to approximate a posterior distribution. Instead, it treats uncertainty estimation as a direct optimization problem using two sub-networks.

Architecture

The NOMU architecture consists of two distinct sub-networks:

  1. $\hat{f}$-network: Responsible for predicting the target value.
  2. $r_{\hat{f}}$-network: Predicts the "raw" model uncertainty.

A key innovation is that the uncertainty network is connected to the final hidden layer of the prediction network. This "Metric Learning" approach allows the uncertainty estimator to reuse the high-level features learned by the predictive model.

The NOMU architecture featuring predictive (f-hat) and uncertainty (r-hat) sub-networks sharing learned features.

The NOMU Loss Function

NOMU utilizes a three-part loss function designed to satisfy specific logical "desiderata":

  • Accuracy: A standard regression loss to fit the training data.
  • In-Sample Constraint: Penalizes uncertainty at known training points, forcing the bounds to "pinch" shut where data is known.
  • Out-of-Sample Push: An exponential term that rewards high uncertainty in regions devoid of training data.

By balancing these terms, NOMU learns a smooth uncertainty surface that is zero at data points and grows as the model moves into unexplored territory.

Key Insights and Desiderata

The authors propose five rules (desiderata) that any robust uncertainty method should follow: (1) Non-negativity, (2) In-sample convergence (zero uncertainty at noiseless points), (3) Out-of-sample growth, (4) Metric learning (reusing predictive features), and (5) Data consistency (vanishing uncertainty as data goes to infinity).

Evaluation and Results

NOMU was evaluated across synthetic functions, a generative Bayesian Neural Network test-bed, and real-world regression tasks.

  • Consistency: In the generative test-bed, NOMU consistently ranked among the top performers, avoiding the volatility seen in methods like Hyper Deep Ensembles (HDE).
  • Bayesian Optimization: In high-dimensional (5D to 20D) tasks, NOMU matched or exceeded the performance of GPs and standard NN methods. Its ability to correctly identify unexplored regions prevented it from becoming trapped in local optima.
  • Robustness: Despite its focus on noiseless epistemic uncertainty, NOMU performed competitively on noisy real-world UCI benchmarks, demonstrating significant robustness.

Conclusion

NOMU provides a principled alternative to stochastic uncertainty estimation. By directly optimizing for uncertainty bounds that respect logical constraints, it offers more reliable guidance for exploration and decision-making in data-scarce environments. Future work includes extending this framework to classification and more complex architectures like Transformers.

Summary updated

BibTeX
@inproceedings{heiss2022nomu,
  author = {Jakob Heiss and Jakob Weissteiner and Hanna Wutte and Sven Seuken and Josef Teichmann},
  title = {{NOMU: Neural Optimization-based Model Uncertainty}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=S42gyD7GRV},
}
Figure