Abstract
Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like models of which MC-Dropout and Deep Ensembles are extreme examples. The first one uses effectively infinite number of highly correlated models while the second one relies on a finite number of independent models. To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of binary masks, which are parameterized in a way that allows to change correlations between individual models. Namely, by controlling the overlap between the masks and their size one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.
Introduction
Deep neural networks are notoriously overconfident, often making incorrect predictions with high certainty. For deployment in safety-critical applications, reliable uncertainty estimation—the ability for a model to "know when it doesn't know"—is essential. Currently, practitioners face a stark trade-off. Deep Ensembles provide the gold standard for reliability by averaging multiple independent models, but at a prohibitive computational cost. Conversely, MC-Dropout offers a cheaper alternative by randomly dropping neurons during inference, yet it often yields highly correlated and unreliable uncertainty estimates.
The Masksembles Framework
Masksembles bridges this gap by viewing uncertainty methods as a continuous spectrum. The core innovation is the Masksembles layer, which replaces standard dropout with a fixed pool of $N$ pre-defined binary masks. The system's behavior is controlled by three primary parameters:
- $N$ (Number of masks): Defines the size of the virtual ensemble.
- $M$ (Model capacity): The number of active units in each mask.
- $S$ (Scale): Controls the overlap (Intersection over Union) between masks.
By tuning the scale $S$, one can explicitly manage the correlation between ensemble members. High $S$ values minimize overlap, forcing the model to learn diverse, independent representations similar to a Deep Ensemble. Low $S$ values increase similarity, mimicking standard dropout or a single model.
Experimental Evaluation and Results
The authors evaluated Masksembles on standard benchmarks including CIFAR-10 and ImageNet, as well as their corrupted variants (CIFAR-10-C, ImageNet-C) to test robustness under domain shift.
Key findings include:
- Performance Parity: On CIFAR-10, Masksembles achieved an Expected Calibration Error (ECE) of 0.01, matching Deep Ensembles while being 2.5x faster to train.
- Scalability: On ImageNet with ResNet-50, Masksembles matched Ensemble performance while being 4x more efficient in training time and memory.
- Robustness: Masksembles significantly outperformed MC-Dropout and single models in maintaining high accuracy and low calibration error as image corruption severity increased.
Conclusion
Masksembles offers a structured approach to managing model diversity without the overhead of full ensembling. By enforcing a specific, low-overlap structure, it forces the network to learn truly diverse sub-networks. While it requires some architectural tuning (adjusting layer width to $M \times S$), it provides a flexible tool for practitioners to navigate the trade-off between computational efficiency and uncertainty reliability. Future work may involve optimizing the mask structure dynamically to further enhance efficiency.
Summary updated