The fundamental promise of ensemble learning is the "wisdom of the crowd": the idea that a collection of diverse models will cancel out each other's idiosyncratic errors, leaving only the correct signal. However, this statistical benefit relies entirely on the assumption that models fail independently. When models share the same training data, architectures, or optimization paths, they tend to fail on the same inputs in the same ways. This phenomenon, known as error correlation, creates an "algorithmic monoculture" where ensembles become redundant, offering little more than a single model at a much higher computational cost.
The Failure of Collective Intelligence
In a perfect ensemble, the probability of the group being wrong decreases as more members are added. As Abels et al. explain, collective intelligence relies on the independence of errors among group members. If errors are uncorrelated, simple aggregation methods like majority voting can effectively filter out noise.
In practice, this independence is rarely achieved. Sarna et al. quantified these correlations across modern machine learning models, finding that error correlations frequently exceed 0.7. This high degree of overlap means that when one model fails, there is a high probability that every other model in the ensemble will fail simultaneously. Kim et al. observed that as Large Language Models (LLMs) become larger and more accurate, their errors actually become more similar, suggesting that scaling alone does not solve the diversity problem.
The Roots of Redundancy
Why do models that are trained separately end up so similar? Research suggests several systemic drivers:
- Shared Lineage: Many modern models are fine-tuned from the same pre-trained checkpoints. Neyshabur et al. demonstrated that pre-training anchors models in a specific optimization basin, leading to highly correlated internal representations and failure modes.
- Data Overlap: When models are trained on the same massive datasets, they learn the same biases. Radharapu et al. noted that if constituent models share identical systematic blind spots, an ensemble lacks the necessary signal to identify and correct those specific failure modes.
- Algorithmic Homogeneity: Using the same mathematical scoring functions or architectures leads to redundant information capture. Aghazade et al. found that fusing retrieval models fails to improve performance if the models produce highly overlapping result sets, even if their underlying mathematical logic differs slightly.
Measuring and Maximizing Diversity
To build effective ensembles, researchers have moved toward explicitly measuring and optimizing for "error diversity" rather than just individual accuracy.
Liu et al. proposed a metric called focal diversity, which measures how well models compensate for each other's specific failures. This approach prioritizes selecting ensemble members that exhibit uncorrelated errors over those with the highest average accuracy. Similarly, Amir et al. introduced a "Uniqueness Score" to identify and replace ensemble members that share failure modes, which significantly enhances robustness against adversarial attacks.
Beyond selection, some methods bake diversity directly into the training process. Manning et al. developed Positive Diversity Tuning (PDT), which explicitly rewards models for being both accurate and different from their peers. In the realm of optimization, Huang et al. introduced Snapshot Ensembles, using cyclic learning rates to force a single model to visit multiple diverse local minima, capturing idiosyncratic mistakes that can be averaged out at test time.
The Path to Robustness
The consensus across recent work is that the "more is better" approach to ensembling has reached a point of diminishing returns. As Fishel et al. and Waterhouse et al. argue, without explicit mechanisms to ensure model diversity, ensembles remain redundant.
The future of the field appears to lie in heterogeneous ensembles. This might involve combining different architectures, using "sidekick" models of different scales as proposed by the authors of the Asymmetric Duos paper, or even creating hybrid crowds of humans and LLMs to inject cognitive diversity that machines currently lack (Abels et al.). By focusing on the independence of failures, practitioners can build systems that are truly more than the sum of their parts.