Randomized ReLU Activation for Uncertainty Estimation of Deep Neural Networks

Yufeng Xia, Jun Zhang, Zhiqiang Gong, Tingsong Jiang, Wen Yao

TL;DR

This paper introduces RBUE, a novel method for estimating uncertainty in deep neural networks by injecting randomness into the ReLU activation function. RBUE achieves comparable uncertainty estimation quality to Deep Ensembles but with significantly reduced computational cost and memory footprint, outperforming MC-Dropout in diversity and reliability.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful tools for learning complex data representations, but a critical challenge remains: reliably assessing the uncertainty of their predictions. Knowing when a model is unsure is crucial for applications like autonomous driving or medical diagnosis. While Deep Ensembles offer high-quality uncertainty estimates, they are computationally prohibitive due to training multiple independent models. Monte Carlo Dropout (MC-Dropout) is a more efficient alternative, but its predictions often lack diversity, leading to less reliable uncertainty estimates. This paper aims to bridge this gap by proposing a method that offers the high-quality uncertainty estimation of Deep Ensembles with the computational efficiency of MC-Dropout.

Methods

The core innovation of this paper is the ReLU-Based Uncertainty Estimation (RBUE) method. Instead of modifying network architecture by dropping neurons (like MC-Dropout) or training multiple full models (like Deep Ensembles), RBUE introduces randomness directly into the ReLU activation function. This is achieved by making the slope of the negative part of the ReLU function a random variable.

Two specific strategies are proposed:

  • MC-DropReLU: This strategy randomly "drops" the ReLU activation with a certain probability q for each neuron. When dropped, the activation becomes linear; otherwise, it remains a standard ReLU. This is akin to a probabilistic masking of the activation function.
  • MC-RReLU: This strategy uses a Random Rectified Linear Unit (RReLU), where the slope of the negative part of the ReLU is sampled from a uniform distribution.

During inference, the network is run multiple times with these randomized activations, generating diverse predictions. The uncertainty is then estimated from the variance or entropy of these predictions, similar to MC-Dropout. The key advantage is that this modification is minimal, only affecting the activation function, and does not require changing the network's overall structure or training procedure significantly.

Datasets

The proposed RBUE method was evaluated on the following widely used datasets:

  • CIFAR10: A dataset of 60,000 32x32 color images in 10 classes.
  • CIFAR100: A dataset of 60,000 32x32 color images in 100 classes.
  • TinyImageNet: A dataset of 120,000 64x64 color images in 200 classes.

Additionally, the authors used corrupted versions of CIFAR10 and TinyImageNet (CIFAR10-C and TinyImageNet-C) to evaluate the models' performance under out-of-distribution conditions, simulating real-world scenarios with noisy or altered inputs.

Key Insights and Ideas

  • Randomness in Activation Functions is Key: The paper demonstrates that injecting randomness directly into the activation function (specifically ReLU) is a viable and effective strategy for generating diverse predictions, which is crucial for reliable uncertainty estimation.
  • Bridging the Gap: RBUE successfully bridges the performance gap between computationally expensive Deep Ensembles and less diverse MC-Dropout, offering a compelling trade-off.
  • Minimal Modification, Maximum Impact: The simplicity of modifying only the activation function makes RBUE easy to implement and integrate into existing DNN architectures without significant architectural changes.
  • Hyperparameter Tuning for Behavior Span: The hyperparameter q in MC-DropReLU allows for a flexible control over the degree of randomness, enabling a spectrum of behaviors from close to MC-Dropout to approaching Deep Ensemble performance.
  • Diversity Matters for Reliability: The experimental analysis strongly supports the hypothesis that increased prediction diversity leads to better uncertainty estimation quality, especially under distributional shifts.

Key Results

  • Competitive Performance: RBUE methods (MC-DropReLU and MC-RReLU) achieve accuracy and Expected Calibration Error (ECE) comparable to Deep Ensembles across CIFAR10, CIFAR100, and TinyImageNet datasets.
  • Significant Cost Savings: RBUE methods require substantially less training time and memory compared to Deep Ensembles (e.g., 4x less storage space). Their computational cost is only slightly higher than MC-Dropout.
  • Improved Diversity over MC-Dropout: Both JSD and DIS metrics show that RBUE methods generate more diverse predictions than MC-Dropout, leading to more reliable uncertainty estimates.
  • Robustness to Corruptions: Under corrupted datasets (distributional shifts), RBUE methods maintain competitive accuracy and ECE, often outperforming MC-Dropout and SVI, and performing on par with Deep Ensembles.
  • Hyperparameter Sensitivity: While generally robust, the optimal q value for MC-DropReLU can vary slightly depending on the model architecture and dataset.

Limitations and Future Work

  • Computational Overhead vs. MC-Dropout: While significantly more efficient than Deep Ensembles, RBUE methods incur a slightly higher training time than MC-Dropout due to the sampling process within the activation function.
  • Optimal Configuration: Determining the optimal placement and configuration of RBUE layers (e.g., which layers to modify) can require empirical tuning, although the paper provides some analysis suggesting that more layers generally lead to better diversity and calibration.

The authors suggest future work will focus on applying RBUE to tasks requiring scalable uncertainty estimation, such as active learning and out-of-distribution detection. They also aim to further explore the trade-offs between computational cost, accuracy, and uncertainty estimation quality by fine-tuning the RBUE hyperparameters.

Summary updated

Abstract

Deep neural networks (DNNs) have successfully learned useful data representations in various tasks. However, assessing the reliability of these representations remains a challenge. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty with higher quality, but it is very expensive to train and test. MC-Dropout is another popular method, which is less expensive but lacks the diversity of predictions. To estimate the uncertainty with higher quality in less time, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of randomly dropping some neurons of the network as in MC-Dropout or using the randomness of the initial weights of networks as in Deep Ensemble, RBUE adds randomness to the activation function module, making the outputs diverse. Under the method, we propose two strategies, MC-DropReLU and MC-RReLU, to estimate uncertainty. We analyze and compare the output diversity of MC-Dropout and our method from the variance perspective and obtain the relationship between the hyperparameters and predictive diversity in the two methods. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time and memory requirements.
BibTeX
@misc{xia2021randomized,
  author = {Yufeng Xia and Jun Zhang and Zhiqiang Gong and Tingsong Jiang and Wen Yao},
  title = {{Randomized ReLU Activation for Uncertainty Estimation of Deep Neural Networks}},
  year = {2021},
  eprint = {2107.07197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.07197},
}
Figure