The Limits of Ensemble Scaling

49 concepts39 papers3 proposals

Ensembling is one of the most reliable strategies in machine learning: train multiple models, average their predictions, and watch the error rate drop. This "wisdom of the crowd" works because individual models make independent errors that cancel each other out.

But what happens when we keep scaling the ensemble? Intuition suggests performance should keep improving, or at least flatten out into a harmless plateau. In reality, recent research reveals a more complex phenomenon: ensemble scaling saturation. Beyond a surprisingly small threshold, adding more models yields diminishing returns, and in many cases, actively degrades performance.

The Exhaustion of the Diversity Budget

The fundamental engine of an ensemble is diversity. If two models make identical predictions, combining them provides zero new information. As modern neural networks grow larger and more capable, they enter what Ryan Theisen et al. describe as the "interpolation regime"—a state where high-capacity models achieve near-zero training error. In When are ensembles really effective?, the authors prove that entering this regime causes different model hypotheses to converge, drastically reducing their disagreement-to-error ratio and creating a regime of diminishing returns.

This convergence means that large ensembles quickly exhaust their "diversity budget." As Christopher D Manning et al. observe in Positive Diversity Tuning for Machine Translation System Combination, the parameter space for unique, high-quality hypotheses is depleted as the number of systems grows. Once an ensemble captures the maximum available signal, new members simply replicate existing predictions rather than correcting systematic gaps.

This lack of cognitive diversity explains why adding more models plateaus so quickly compared to human groups, who benefit from independent error patterns (Axel Abels et al.). When training causes different model hypotheses to become nearly identical, the benefits of averaging across multiple ensemble members disappear entirely (David Pape et al.). Furthermore, Ekaterina Lobacheva et al. demonstrate in On Power Laws in Deep Ensembles that as individual neural networks grow larger, the relative performance gain from ensembling decreases because large models naturally converge toward similar solutions.

Aggregation Complexity and Combination Loss

The limitation isn't just that new models are redundant; it is also that combining them is fundamentally lossy. Traditional ensemble theories often treat the aggregation step as a trivial average, failing to explain why performance plateaus. To address this, Gaku Morio introduced a new theoretical framework in Rethinking Fano's Inequality in Ensemble Learning.

The author models ensemble scaling as a direct trade-off between predictive information and combination loss—the information lost when merging multiple model predictions. When you add a base learner to an ensemble, you hope to increase the collective information about the target. However, each additional model also increases the complexity of the aggregation task. The central judge or voting mechanism must now filter through overlapping, highly correlated, or conflicting outputs. As the ensemble size grows, the incremental gains in collective information are eventually completely outweighed by the rising information loss during the combination phase.

When More Models Introduce More Noise

When an ensemble scales past its saturation point, performance does not always just plateau—it can actively decline. This occurs because large committees introduce noise and redundancy that dilute the contributions of the highest-quality participants.

In multi-LLM collaborative environments, Junhua Fang et al. found that increasing the number of models in a committee beyond a small threshold introduces noise that dilutes high-performing models rather than providing additive benefits (Multi-LLM Text Summarization). A similar phenomenon occurs in open-ended reasoning tasks: Haonan Wang et al. note that running too many parallel reasoning trajectories can degrade performance because low-quality trajectories dilute the correct logical signals.

This injection of noise is a shared vulnerability across modalities. In computer vision, Akis Linardos et al. observed that increasing the number of diverse backbones in a saliency model eventually introduces more noise than useful signal, harming final predictions (DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling).

Finding the Practical Threshold

Empirical evidence across diverse architectures points to a remarkably low saturation threshold, showing that the vast majority of an ensemble's power is unlocked almost immediately:

Understanding these scaling limits shifts the core engineering challenge. Instead of blindly allocating compute to expand ensemble size, the goal must be to carefully manage the trade-off between individual model optimization and collective diversity.

Key Foundational Work

Large-scale prediction of long disordered regions in proteins using random forests (2009) — contributed 1 concept elements to this direction.
An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems (2007) — contributed 1 concept elements to this direction.
On the selection of decision trees in Random Forests (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ensemble_scaling_saturation_a2e88f63,
  title = {ensemble scaling saturation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-scaling-saturation/rdr_a2e88f63c65c460b83c6c8e5e22fb226},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_ensemble_scaling_saturation_a2e88f63_v1,
  title = {ensemble scaling saturation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-scaling-saturation/rdr_a2e88f63c65c460b83c6c8e5e22fb226/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake