Deep ensembles are the gold standard for uncertainty quantification, calibration, and adversarial robustness. By training multiple independent neural networks, we can capture epistemic uncertainty—what a model doesn't know—by measuring how much individual members disagree. As shown by Kaizheng Wang et al. in Credal Deep Ensembles for Uncertainty Quantification, this diversity allows models to predict robust probability intervals rather than overconfident single values. This property is incredibly powerful: it helps prevent overconfident errors in visual question answering (Mahsa Mozaffari et al. in GLEN: Generalized Focal Loss Ensemble of Low-Rank Networks for Calibrated Visual Question Answering) and acts as a robust defense against black-box adversarial attacks by injecting query uncertainty (Quoc Viet Vo et al. in Can Model Randomization Offer Robustness Against Query-Based Black-Box Attacks?).
However, this performance comes with a severe penalty: the multi-model tax. Standard deep ensembles require training, storing, and executing $N$ separate models. This linear scaling of resource consumption introduces massive computational and memory overhead during both training and inference, making them impractical for resource-constrained hardware (Peter R. D. van der Wal et al. in Multilayer perceptron ensembles in a truly sparse training context). For instance, in model-based offline reinforcement learning, evaluating candidate states through an ensemble of discriminators dramatically increases training duration and memory consumption (Fan-Ming Luo et al. in Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning). Similarly, executing multiple models simultaneously during inference creates a severe scalability bottleneck (Md Ibrahim Ibne Alam et al. in On the Utility of Existing Fine-Tuned Models on Data-Scarce Domains).
To make ensembles practical for edge devices and large-scale production environments, researchers are developing techniques to bypass this $N\times$ cost bottleneck.
Native Sparsity and Low-Rank Compression
One intuitive way to reduce the ensemble tax is to make each individual member smaller. However, simply applying binary masks to simulate sparsity on dense matrices doesn't actually save compute on standard hardware.
To solve this, Peter R. D. van der Wal et al. developed Truly Sparse Ensembles in Multilayer perceptron ensembles in a truly sparse training context. Instead of simulated sparsity, the authors utilize native sparse data structures to train independent subnetworks from scratch. By avoiding dense matrix operations entirely, they achieve the accuracy of dense ensembles while drastically reducing FLOPs and training time on standard CPU hardware.
Another angle is compressing specific layers where redundancy is highest. In GLEN: Generalized Focal Loss Ensemble of Low-Rank Networks for Calibrated Visual Question Answering, Mahsa Mozaffari et al. applied low-rank matrix factorization to the final layers of the network. Combined with a specialized focal loss to enforce diversity, this approach minimizes the parameter footprint of each ensemble member while maintaining robust calibration.
Virtualizing Ensembles inside a Single Parameter Space
What if we could eliminate the need for multiple distinct models entirely? Beyond hardware costs, traditional ensembles introduce significant operational complexity in terms of training, versioning, and deploying multiple distinct models in production (Jun Suzuki et al. in Single Model Ensemble using Pseudo-Tags and Distinct Vectors).
To address this, Jun Suzuki et al. developed a method called SINGLEENS in Single Model Ensemble using Pseudo-Tags and Distinct Vectors. The intuition is to partition a single parameter space into distinct, virtual subspaces. By appending unique "pseudo-tags" and routing inputs through orthogonal vectors, the model learns to simulate multiple independent networks. This parameter virtualization achieves ensemble-level accuracy with up to a 9x reduction in parameters, completely bypassing the multi-model administrative and hardware overhead.
Amortizing Costs via Shared Latents and Mutual Learning
Another major bottleneck occurs when ensembles are used dynamically within larger systems, such as reinforcement learning loops. In model-based reinforcement learning, agents use ensembles of world models to plan optimistically under uncertainty. However, processing raw environment states through multiple independent networks is incredibly expensive.
In Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles, Tim Seyde et al. mitigated this by sharing a latent representation layer across the entire ensemble. Because the heavy lifting of feature extraction from raw inputs is done once, the ensemble members only need to predict next steps within a compact latent space, significantly lowering the computational barrier to scaling up complex robotic tasks.
Alternatively, we can accept high computational costs during the training phase if it guarantees an efficient single-model deployment at inference. In Improving Accuracy and Calibration via Differentiated Deep Mutual Learning, Han Liu et al. introduced Differentiated Deep Mutual Learning (Diff-DML). During training, multiple models run concurrently and use asymmetric strategies to force each other to remain diverse. While this simultaneous optimization increases training-time overhead (Ancong Wu et al. in Single-modality self-supervised information mining for cross-modality person re-identification), it allows the final deployed model to inherit the calibration and accuracy benefits of the heavy ensemble without paying any computational tax at inference.
Key Foundational Work
Bagging and Boosting a Treebank Parser (2000) — contributed 1 concept elements to this direction.
Heterogeneity and Inaccuracy in Protein Structures Solved by X-Ray Crystallography (2004) — contributed 1 concept elements to this direction.