Masksembles for Uncertainty Estimation

Nikita Durasov, Timur Bagautdinov, Pierre Baqué, Pascal Fua

Abstract

Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like models of which MC-Dropout and Deep Ensembles are extreme examples. The first one uses effectively infinite number of highly correlated models while the second one relies on a finite number of independent models. To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of binary masks, which are parameterized in a way that allows to change correlations between individual models. Namely, by controlling the overlap between the masks and their size one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.

Introduction

Deep neural networks are notoriously overconfident, often making incorrect predictions with high certainty. For deployment in safety-critical applications, reliable uncertainty estimation—the ability for a model to "know when it doesn't know"—is essential. Currently, practitioners face a stark trade-off. Deep Ensembles provide the gold standard for reliability by averaging multiple independent models, but at a prohibitive computational cost. Conversely, MC-Dropout offers a cheaper alternative by randomly dropping neurons during inference, yet it often yields highly correlated and unreliable uncertainty estimates.

The Masksembles Framework

Masksembles bridges this gap by viewing uncertainty methods as a continuous spectrum. The core innovation is the Masksembles layer, which replaces standard dropout with a fixed pool of $N$ pre-defined binary masks. The system's behavior is controlled by three primary parameters:

  • $N$ (Number of masks): Defines the size of the virtual ensemble.
  • $M$ (Model capacity): The number of active units in each mask.
  • $S$ (Scale): Controls the overlap (Intersection over Union) between masks.

By tuning the scale $S$, one can explicitly manage the correlation between ensemble members. High $S$ values minimize overlap, forcing the model to learn diverse, independent representations similar to a Deep Ensemble. Low $S$ values increase similarity, mimicking standard dropout or a single model.

The spectrum of uncertainty estimation methods, showing Masksembles as an optimal configuration between single models and full ensembles.

Experimental Evaluation and Results

The authors evaluated Masksembles on standard benchmarks including CIFAR-10 and ImageNet, as well as their corrupted variants (CIFAR-10-C, ImageNet-C) to test robustness under domain shift.

Key findings include:

  • Performance Parity: On CIFAR-10, Masksembles achieved an Expected Calibration Error (ECE) of 0.01, matching Deep Ensembles while being 2.5x faster to train.
  • Scalability: On ImageNet with ResNet-50, Masksembles matched Ensemble performance while being 4x more efficient in training time and memory.
  • Robustness: Masksembles significantly outperformed MC-Dropout and single models in maintaining high accuracy and low calibration error as image corruption severity increased.
Comparison of Expected Calibration Error (ECE) and Accuracy across different corruption severities for various methods.

Conclusion

Masksembles offers a structured approach to managing model diversity without the overhead of full ensembling. By enforcing a specific, low-overlap structure, it forces the network to learn truly diverse sub-networks. While it requires some architectural tuning (adjusting layer width to $M \times S$), it provides a flexible tool for practitioners to navigate the trade-off between computational efficiency and uncertainty reliability. Future work may involve optimizing the mask structure dynamically to further enhance efficiency.

Summary updated

Abstract

Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like models of which MC-Dropout and Deep Ensembles are extreme examples. The first uses an effectively infinite number of highly correlated models while the second relies on a finite number of independent models. To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of binary masks, which are parameterized in a way that allows to change correlations between individual models. Namely, by controlling the overlap between the masks and their density one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.
BibTeX
@inproceedings{durasov2021masksembles,
  author = {Nikita Durasov and Timur Bagautdinov and Pierre Baqué and Pascal Fua},
  title = {{Masksembles for Uncertainty Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rX9ooBq1D1a},
}
Figure