Deep ensembles are one of the most reliable tools in machine learning for improving generalization, calibrating uncertainty, and ensuring adversarial robustness. By averaging the predictions of multiple independently trained neural networks, ensembles smooth out the idiosyncratic errors of single models.
However, this performance boost comes with a severe penalty: training $N$ independent models multiplies the required computational budget, memory footprint, and training time by $O(N)$. As deep learning models scale to billions of parameters, this linear overhead becomes a prohibitive bottleneck.
To resolve this dilemma, a prominent research direction focuses on low-overhead ensembling. Instead of treating every ensemble member as an entirely isolated training run, these methods exploit the underlying geometry of neural network loss surfaces, harvest intermediate states from a single training trajectory, or share parameters to achieve the benefits of diversity at a fraction of the cost.
Exploiting Loss Surface Geometry
The traditional view of neural network optimization imagines a loss surface dotted with isolated, low-loss "islands" separated by high-energy barriers. Under this mental model, the only way to find diverse solutions is to restart the optimization process from scratch with different random initializations.
However, research into the geometry of loss surfaces reveals a different reality: local optima are often connected by continuous, high-accuracy "valleys"—a phenomenon known as mode connectivity.
Standard Ensembling:
[Init 1] ----> (Optimum 1)
[Init 2] ----> (Optimum 2) (Isolated paths, full training cost x N)
Fast Geometric Ensembling (FGE):
[Init] ----> (Optimum 1) ~~~[LR Cycle]~~~> (Optimum 2) ~~~> (Optimum 3)
(Traversing a shared low-loss valley)
By leveraging this geometric insight, Garipov et al. introduced Fast Geometric Ensembling (FGE). Instead of restarting optimization, FGE uses a cyclical learning rate schedule to force the optimizer to travel along these low-loss valleys. As the learning rate spikes and decays, the model bounces out of its immediate local minimum but stays within the high-accuracy valley, allowing researchers to collect a diverse set of high-performing models in the time it typically takes to train just a single network.
Taking this concept a step further, Wortsman et al. proposed learning entire subspaces of neural networks. Instead of optimizing individual point weights, their method parameterizes a geometric subspace—such as a line, curve, or simplex—directly within the loss surface. By optimizing the parameters defining the subspace so that every point within it achieves low loss, a single training run yields an infinite family of models. The endpoints of these subspaces provide functionally diverse solutions that can be ensembled at no extra training cost.
Harvesting the Training Trajectory
If the goal is to find multiple distinct weight configurations that perform well, we do not necessarily need to look across different valleys; we can look backward along our own path. A model's optimization trajectory contains a wealth of historical information that is usually discarded at the end of training.
One approach is to treat intermediate training checkpoints as a ready-made ensemble. For instance, Chen et al. introduced Self-Ensemble Protection (SEP), which repurposes checkpoints from a single training run to form a diverse "self-ensemble." Because checkpoints from different epochs have observed the data at varying stages of optimization, they exhibit enough functional diversity to generate robust data perturbations without the overhead of training independent networks.
Alternatively, rather than keeping the checkpoints separate and averaging their outputs at inference time, we can average their weights during training. Arpit and Wang developed the Ensemble of Averages (EoA) protocol, which maintains moving averages of model parameters across the training trajectory. This stabilizes the optimization path and provides the generalization benefits of an ensemble while collapsing the final deployment back into a single, computationally efficient model.
Amortizing Architecture via Parameter Sharing
When independent models are trained from scratch, they waste significant computation relearning identical low-level features (such as edges and textures in images). We can eliminate this redundancy by forcing models to share a common structural backbone while maintaining distinct "views" or sub-networks.
This structural amortization can happen at the layer level or the network level:
- Multi-Task Subnetworks: In semi-supervised learning, traditional bootstrapping methods like tri-training require maintaining three entirely separate models to generate pseudo-labels. To mitigate this, Ruder and Plank introduced Multi-task Tri-training (MT-Tri), which shares the lower layers of the neural network across all three models, drastically reducing memory consumption and accelerating training.
- Virtual Ensembling: Instead of explicitly instantiating multiple networks, we can generate an ensemble on-the-fly through dynamic perturbations. Bai et al. developed Ghost Networks, which apply random feature-level perturbations to a single base network during the forward pass. This creates a massive, virtual ensemble of distinct network paths on-the-fly, bypassing the memory bottleneck of maintaining multiple high-performing models in hardware.
Compressing Ensembles via Weight Merging
Even if we find efficient ways to train an ensemble, running multiple models at inference time still incurs a linear cost in latency and memory. A recent breakthrough paradigm shifts the focus from ensembling predictions to merging weights.
Prediction Ensembling (Expensive Inference):
Input ---> [Model A] ---> Probabilities \
Input ---> [Model B] ---> Probabilities ---> Average ---> Output
Weight Merging (Efficient Inference):
[Model A] \
---> Analytical Merge ---> [Merged Model] ---> Input ---> Output
[Model B] /
Traditional weight-averaging methods (like Model Soups) require training and storing dozens of independent models to find a good combination. Jang addressed this bottleneck by introducing Model Stock, a highly efficient weight-merging technique.
Jang uncovered a unique geometric property of fine-tuning: when multiple models are fine-tuned from the same pre-trained baseline, their weights do not scatter randomly. Instead, they reside on a "thin shell" surrounding a centralized, optimal parameter configuration. By exploiting this spherical geometry, Model Stock uses a simple mathematical formula to calculate the exact center of this shell using as few as two fine-tuned models. This achieves the accuracy and robustness of a massive ensemble while being up to 24x more computationally efficient during the preparation phase, and requiring only a single model's cost at inference time.