SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates

Lingkai Kong, Jimeng Sun, Chao Zhang

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in various domains but often suffer from overconfidence, particularly when encountering out-of-distribution (OOD) data. This lack of reliable uncertainty estimation poses significant risks in safety-critical applications such as autonomous driving and medical diagnostics. Existing approaches, such as Bayesian Neural Networks (BNNs) and Deep Ensembles, offer solutions but are often limited by computational complexity or difficulties in defining appropriate priors.

This paper introduces SDE-Net, a novel framework that treats deep learning as a stochastic dynamical system. By incorporating Brownian motion into the network's architecture, SDE-Net explicitly quantifies uncertainty by separating predictive "drift" from "diffusion" caused by model ignorance.

The SDE-Net Framework

SDE-Net is inspired by the mathematical connection between Residual Networks (ResNets) and Ordinary Differential Equations (ODEs). In a standard ResNet, the forward pass can be viewed as a state evolving over time through discrete layers. SDE-Net extends this concept to a continuous-time Stochastic Differential Equation (SDE):

$$dx_t = f(x_t, t)dt + g(x_t, t)dW_t$$

The model comprises two primary components:

  1. The Drift Net ($f$): This component represents the deterministic part of the system, responsible for mapping inputs to predictive labels. It captures aleatoric uncertainty—the inherent noise in the data—by outputting a probability distribution. In regions of high data density, the drift net dominates the dynamics.
  2. The Diffusion Net ($g$): This component represents the stochastic term (Brownian motion). Its role is to capture epistemic uncertainty, or model ignorance. When the model encounters OOD data, the diffusion net increases the variance of the hidden states. This effectively "shakes" the internal state, causing the final prediction to be scattered across multiple forward passes, thereby signaling high uncertainty.
Predictive distributions for in-distribution data showing localized, high-confidence regions.

Training and Regularization

The training objective for SDE-Net is designed to enforce the separation of drift and diffusion. The model minimizes standard prediction error while keeping diffusion low for in-distribution samples.

Crucially, the authors introduce a regularization term that explicitly maximizes diffusion for OOD samples. Since true OOD data is not available during training, the authors synthesize "pseudo-OOD" samples by adding Gaussian noise to the training data. This ensures that the model learns to increase its diffusion term when it detects patterns that deviate from the training distribution. This mechanism acts as a "safety valve," preventing the model from making confident but incorrect predictions on unfamiliar inputs.

Predictive distributions for out-of-distribution data showing increased variance and diffusion.

Experimental Results

SDE-Net was evaluated across multiple benchmarks, demonstrating state-of-the-art performance in OOD detection and active learning.

  • OOD Detection: In classification tasks (e.g., MNIST vs. SEMEION), SDE-Net achieved an AUROC of 99.9%, surpassing Deep Ensembles (98.8%) and MC-Dropout (97.6%). The model's ability to distinguish between seen and unseen classes was consistently superior across SVHN and CIFAR datasets.
  • Regression Breakthrough: On the Year Prediction MSD dataset, SDE-Net achieved an OOD AUROC of 84.4%. This is a significant improvement over traditional baselines like Deep Ensembles and BNNs, which typically perform near random guess levels (~50-60%) on this challenging task.
  • Active Learning: The model's uncertainty estimates proved highly effective for active learning. By identifying the most "uncertain" points for human labeling, SDE-Net reduced model error faster than traditional Bayesian methods as the training set size increased.

Conclusion

SDE-Net provides a scalable and theoretically grounded approach to uncertainty estimation in deep learning. By leveraging the properties of stochastic dynamical systems, it achieves superior OOD detection with lower computational overhead than ensemble methods. While future work is needed to enhance robustness against sophisticated adversarial attacks and reduce inference latency, SDE-Net represents a major step toward safer and more reliable AI systems.

Summary updated

BibTeX
@inproceedings{kong2020sde,
  author = {Lingkai Kong and Jimeng Sun and Chao Zhang},
  title = {{SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=k7Qk7gaOUtn},
}
Figure