Masksembles for Uncertainty Estimation

Nikita Durasov, Timur M. Bagautdinov, Pierre Baqué, Pascal Fua

Abstract

Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like models of which MC-Dropout and Deep Ensembles are extreme examples. The first uses effectively infinite number of highly correlated models while the second relies on a finite number of independent models. To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of binary masks, which are parameterized in a way that allows to change correlations between individual models. Namely, by controlling the overlap between the masks and their density one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.

Introduction

Deep neural networks are notoriously overconfident, often providing definitive answers even when incorrect. To deploy these models safely in high-stakes environments like autonomous driving or medical diagnosis, accurate uncertainty estimation is critical. Currently, practitioners face a trade-off between the efficiency of MC-Dropout—which often underestimates uncertainty due to highly correlated predictions—and the reliability of Deep Ensembles, which are computationally expensive to train and maintain.

Masksembles addresses this "missing middle" by treating uncertainty estimation as a continuous spectrum. By utilizing a fixed set of structured binary masks with controllable overlap, Masksembles achieves the diversity of an ensemble without the massive overhead of training entirely separate networks.

Methods: Bridging the Ensemble Spectrum

Masksembles replaces the unstructured randomness of standard dropout with a fixed set of $N$ pre-defined binary masks. These masks are generated using three key hyperparameters: the number of masks ($N$), the number of active features ($M$), and a scale parameter ($S$) that controls the overlap between masks.

The spectrum of ensemble-like models. Masksembles allows practitioners to navigate between Single Models, Ensembles, and MC-Dropout by tuning the number of masks $N$ and the scale $S$.

During training, the model randomly selects one of the $N$ masks for each sample. At inference, the model runs $N$ forward passes, each using one of the fixed masks, and averages the results to produce a final prediction and uncertainty estimate. By adjusting the scale $S$, the correlation between these implicit "sub-models" can be tuned. A low $S$ results in heavy overlap (similar to MC-Dropout), while a high $S$ creates disjoint masks that force different parts of the network to learn independent representations, mimicking a Deep Ensemble.

Key Insights and Results

The core innovation of Masksembles is the concept of "structured redundancy." By increasing layer size slightly and applying disjoint masks, multiple "sub-models" can be embedded within a single backbone. This provides ensemble-like benefits while sharing early-layer computations.

The method was evaluated on CIFAR-10 and ImageNet, including corrupted versions (CIFAR-10-C and ImageNet-C) to test robustness under distribution shift.

Expected Calibration Error (ECE) across different corruption severities. Masksembles consistently matches or exceeds the calibration performance of Deep Ensembles while significantly outperforming MC-Dropout.
  • Performance vs. Cost: On CIFAR-10, Masksembles matched the accuracy and calibration of Deep Ensembles while being nearly twice as memory-efficient. On ImageNet, it achieved ensemble-level performance with a training footprint 4x smaller than traditional ensembles.
  • Robustness: Masksembles consistently outperformed MC-Dropout in Out-of-Distribution (OOD) detection and on corrupted images, where MC-Dropout's uncertainty estimates typically break down.
  • Ease of Integration: As a simple masking layer, it can be added to existing architectures like ResNet or Wide-ResNet without altering core logic.

Conclusion

Masksembles provides a scalable and effective solution for uncertainty estimation, bridging the gap between efficiency and reliability. While it significantly reduces training costs, it still requires multiple forward passes at inference time. Future work may explore parallelizing these passes or applying the technique to domains like Reinforcement Learning and Bayesian Optimization.

Summary updated

Abstract

Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like models of which MC-Dropout and Deep Ensembles are extreme examples. The first uses effectively infinite number of highly correlated models while the second relies on a finite number of independent models. To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of binary masks, which are parameterized in a way that allows to change correlations between individual models. Namely, by controlling the overlap between the masks and their density one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.
BibTeX
@inproceedings{durasov2021masksembles,
  author = {Nikita Durasov and Timur M. Bagautdinov and Pierre Baqué and Pascal Fua},
  title = {{Masksembles for Uncertainty Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rX9ooBq1D1a},
}
Figure