Toward Robust Uncertainty Estimation with Random Activation Functions

Yana Stoyanova, Soroush Ghandi, Maryam TavakolAAAI 2023 · DOI

Introduction

Deep learning models are notoriously overconfident, often making "confidently wrong" predictions on data they’ve never seen before. In safety-critical fields like healthcare or autonomous driving, a model's ability to quantify its own uncertainty—knowing what it doesn't know—is just as important as its accuracy. This paper introduces Random Activation Functions (RAFs) Ensemble, a simple but highly effective method that boosts ensemble diversity to provide more robust uncertainty bounds.

The Challenge of Ensemble Diversity

Uncertainty is generally categorized into two types: aleatoric (noise inherent in the data) and epistemic (uncertainty due to a lack of data or knowledge). Ensemble methods—training multiple models and measuring their disagreement—are the gold standard for estimating epistemic uncertainty. However, ensembles often suffer from a lack of diversity. If every network in an ensemble shares the same architecture and is trained on similar data, they tend to converge to similar biased solutions, making the same mistakes and agreeing even when they are wrong.

Random Activation Functions (RAFs) Ensemble

The authors build upon the Anchored Ensemble technique, which regularizes model parameters toward a random initial "anchor" to mimic Bayesian inference. Their novel contribution, the RAFs Ensemble, introduces structural diversity by assigning different activation functions to each member network.

  1. Diverse Architectures: Instead of using a uniform activation function (like ReLU) across the ensemble, each network is assigned a function from a pool of seven: GELU, Softsign, Swish, SELU, Tanh, Error function, and Linear (identity).
  2. Training & Inference: Each network is initialized with random weights and a random activation function, then trained using a regularized loss function. During prediction, the ensemble calculates the mean of all outputs, using the variance between them as the measure of epistemic uncertainty.

By varying the non-linear properties of each network, the ensemble is forced to explore a wider variety of functional "shapes," leading to higher disagreement in regions where data is scarce (Out-of-Distribution).

Experimental Evaluation

The method was evaluated across a wide range of benchmarks:

  • Synthetic Data: 20 datasets designed to test specific challenges, such as functions with many local minima (MLM) and trigonometric functions with "gaps" in the training data to assess OOD performance.
  • Real-World Regression: Standard benchmarks including Boston Housing, Abalone, and Naval Propulsion.
  • Scalability: High-dimensional datasets like Superconductivity (65 features) and Social Network Shares (40 features) to demonstrate performance beyond toy problems.

Key Results and Discussion

The RAFs Ensemble demonstrated significant improvements in both accuracy and uncertainty calibration.

  • NLL Dominance: In terms of Negative Log-Likelihood (NLL), RAFs Ensemble outperformed five major baselines, including Deep Ensembles and Hyperdeep Ensembles, on 13 out of 25 datasets.
  • OOD Robustness: Qualitative analysis shows that while standard ensembles often "collapse" and become overconfident far from training data, RAFs maintains wide, realistic uncertainty bounds.
Qualitative comparison of uncertainty estimation methods on a synthetic gap dataset. RAFs (f) maintains realistic uncertainty in the gap compared to standard ensembles like DE (a) and AE (b).
  • Scalability: On the 65-dimensional Superconductivity dataset, RAFs achieved an NLL of 5.49, significantly outperforming the strongest baseline (RP-param), which failed with an NLL over 100.

Conclusion

RAFs Ensemble offers a "plug-and-play" modification to standard ensemble pipelines that significantly enhances epistemic uncertainty estimation. By introducing architectural variety through random activation functions, the method prevents model convergence to biased solutions in data-sparse regions. While currently focused on regression, extending this approach to large-scale classification tasks remains a promising direction for future research.

Summary updated

BibTeX
@inproceedings{stoyanova2023toward,
  author = {Yana Stoyanova and Soroush Ghandi and Maryam Tavakol},
  title = {{Toward Robust Uncertainty Estimation with Random Activation Functions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26768},
}
Figure