Abstract
Deep neural networks (DNNs) have successfully learned useful data representations in various tasks. However, assessing the reliability of these representations remains a challenge. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty with higher quality, but it is very expensive to train and test. MC-Dropout is another popular method, which is less expensive but lacks the diversity of predictions. To estimate the uncertainty with higher quality in less time, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of randomly dropping some neurons of the network as in MC-Dropout or using the randomness of the initial weights of networks as in Deep Ensemble, RBUE adds randomness to the activation function module, making the outputs diverse. Under the method, we propose two strategies, MC-DropReLU and MC-RReLU, to estimate uncertainty. We analyze and compare the output diversity of MC-Dropout and our method from the variance perspective and obtain the relationship between the hyperparameters and predictive diversity in the two methods. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time and memory requirements.
Introduction
Deep Neural Networks (DNNs) have become ubiquitous in learning data representations, yet they often suffer from overconfidence, providing incorrect predictions with high certainty. In safety-critical domains like autonomous driving or medical diagnosis, reliable uncertainty estimation is paramount. Currently, the field is divided between high-quality but computationally prohibitive methods like Deep Ensembles and more efficient but less diverse methods like MC-Dropout.
RBUE (ReLU-Based Uncertainty Estimation) addresses this "diversity gap" by injecting randomness directly into the activation function. This approach offers a scalable alternative that maintains high-quality uncertainty estimates while avoiding the massive cost of training an entire ensemble.
Methodology: Randomizing the Activation Function
The core innovation of RBUE is shifting the source of stochasticity from neuron weights or connectivity to the activation function itself. Instead of the standard ReLU ($y = \max(0, x)$), the authors propose two variants:
- MC-DropReLU: For negative inputs ($x < 0$), the function randomly decides whether to set the output to zero (standard ReLU) or let the input pass through unchanged (linear), governed by a retention rate $q$.
- MC-RReLU (Random Rectified Linear Unit): The slope for negative inputs is sampled from a uniform distribution $U(l, u)$.
During inference, the model performs $N$ forward passes for a single input. Because the activation functions behave differently in each pass, the model generates a distribution of predictions. The variance or entropy of these predictions provides a robust measure of uncertainty.
Experimental Evaluation
The authors validated RBUE on CIFAR-10, CIFAR-100, and TinyImageNet, including their "corrupted" counterparts (e.g., CIFAR-10-C). These corrupted datasets are essential for evaluating how well a model identifies out-of-distribution data where it should be less certain.
Key findings include:
- Efficiency: MC-DropReLU achieves performance competitive with Deep Ensembles while requiring approximately 4x less memory and significantly reduced training time.
- Calibration: RBUE demonstrated superior Expected Calibration Error (ECE) compared to MC-Dropout, particularly on corrupted data where it sometimes outperformed Deep Ensembles.
- Representation Diversity: Metrics like Jensen-Shannon Divergence (JSD) confirmed that RBUE produces more diverse internal representations than Dropout, leading to more honest uncertainty assessments.
Conclusion
RBUE provides a simple, "drop-in" replacement for standard ReLU layers that significantly improves uncertainty estimation without the overhead of full ensembles. While it still requires multiple forward passes during inference, its training efficiency and calibration quality make it a compelling choice for robust deep learning. Future work may explore its application in active learning and real-time deployment on edge devices.
Summary updated