Ensembling is one of the most reliable strategies in machine learning: train multiple models, average their predictions, and watch the error rate drop. This "wisdom of the crowd" works because individual models make independent errors that cancel each other out.
But what happens when we keep scaling the ensemble? Intuition suggests performance should keep improving, or at least flatten out into a harmless plateau. In reality, recent research reveals a more complex phenomenon: ensemble scaling saturation. Beyond a surprisingly small threshold, adding more models yields diminishing returns, and in many cases, actively degrades performance.
The Exhaustion of the Diversity Budget
The fundamental engine of an ensemble is diversity. If two models make identical predictions, combining them provides zero new information. As modern neural networks grow larger and more capable, they enter what Ryan Theisen et al. describe as the "interpolation regime"—a state where high-capacity models achieve near-zero training error. In When are ensembles really effective?, the authors prove that entering this regime causes different model hypotheses to converge, drastically reducing their disagreement-to-error ratio and creating a regime of diminishing returns.
This convergence means that large ensembles quickly exhaust their "diversity budget." As Christopher D Manning et al. observe in Positive Diversity Tuning for Machine Translation System Combination, the parameter space for unique, high-quality hypotheses is depleted as the number of systems grows. Once an ensemble captures the maximum available signal, new members simply replicate existing predictions rather than correcting systematic gaps.
This lack of cognitive diversity explains why adding more models plateaus so quickly compared to human groups, who benefit from independent error patterns (Axel Abels et al.). When training causes different model hypotheses to become nearly identical, the benefits of averaging across multiple ensemble members disappear entirely (David Pape et al.). Furthermore, Ekaterina Lobacheva et al. demonstrate in On Power Laws in Deep Ensembles that as individual neural networks grow larger, the relative performance gain from ensembling decreases because large models naturally converge toward similar solutions.
Aggregation Complexity and Combination Loss
The limitation isn't just that new models are redundant; it is also that combining them is fundamentally lossy. Traditional ensemble theories often treat the aggregation step as a trivial average, failing to explain why performance plateaus. To address this, Gaku Morio introduced a new theoretical framework in Rethinking Fano's Inequality in Ensemble Learning.
The author models ensemble scaling as a direct trade-off between predictive information and combination loss—the information lost when merging multiple model predictions. When you add a base learner to an ensemble, you hope to increase the collective information about the target. However, each additional model also increases the complexity of the aggregation task. The central judge or voting mechanism must now filter through overlapping, highly correlated, or conflicting outputs. As the ensemble size grows, the incremental gains in collective information are eventually completely outweighed by the rising information loss during the combination phase.
When More Models Introduce More Noise
When an ensemble scales past its saturation point, performance does not always just plateau—it can actively decline. This occurs because large committees introduce noise and redundancy that dilute the contributions of the highest-quality participants.
In multi-LLM collaborative environments, Junhua Fang et al. found that increasing the number of models in a committee beyond a small threshold introduces noise that dilutes high-performing models rather than providing additive benefits (Multi-LLM Text Summarization). A similar phenomenon occurs in open-ended reasoning tasks: Haonan Wang et al. note that running too many parallel reasoning trajectories can degrade performance because low-quality trajectories dilute the correct logical signals.
This injection of noise is a shared vulnerability across modalities. In computer vision, Akis Linardos et al. observed that increasing the number of diverse backbones in a saliency model eventually introduces more noise than useful signal, harming final predictions (DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling).
Finding the Practical Threshold
Empirical evidence across diverse architectures points to a remarkably low saturation threshold, showing that the vast majority of an ensemble's power is unlocked almost immediately:
- Monocular Depth Estimation: In Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One, Ran Li shows that the most significant accuracy improvements occur when moving from one to just three base predictors, after which marginal gains fail to justify the added computational complexity.
- Collaborative Filtering: The authors of Probabilistic Ensemble of Collaborative Filters find that performance typically saturates after the addition of a modest ten to fifteen components.
- Tree-Based Models: In boosted tree ensembles, Alexandre Forel et al. demonstrate that scaling size leads to a skyrocketing proportion of redundant trees, allowing models to be aggressively compressed without any loss in predictive performance (Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles).
Understanding these scaling limits shifts the core engineering challenge. Instead of blindly allocating compute to expand ensemble size, the goal must be to carefully manage the trade-off between individual model optimization and collective diversity.
Key Foundational Work
Large-scale prediction of long disordered regions in proteins using random forests (2009) — contributed 1 concept elements to this direction.
An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems (2007) — contributed 1 concept elements to this direction.
On the selection of decision trees in Random Forests (2009) — contributed 1 concept elements to this direction.