The "wisdom of the crowd" is a foundational intuition in machine learning: if a single model is prone to errors, a committee of models should, in theory, cancel out those errors and converge on the truth. However, recent research into Large Language Models (LLMs) and deep neural networks has revealed a counter-intuitive "Goldilocks" principle. Simply scaling the number of models in an ensemble does not lead to indefinite improvements. Instead, performance often plateaus or even degrades as the committee grows.
This research direction focuses on identifying the optimal size and composition of model collectives, moving away from "naive" ensembling toward strategic selection and sophisticated interaction patterns.
The Limits of Naive Scaling
In traditional ensemble learning, more is usually better. But in the era of high-capacity models, researchers have identified a persistent "performance plateau." Junhua Fang and colleagues found that in multi-LLM summarization, increasing the number of models beyond a small threshold introduces noise and redundancy that dilutes the contributions of high-performing models Multi-LLM Text Summarization.
This phenomenon isn't limited to text. In semi-supervised image segmentation, Jizong Peng and Marco Pedersoli observed that the most significant gains occur when moving from one to two models, with a fourth model offering almost no additional benefit Deep co-training for semi-supervised image segmentation.
Why does this happen? The papers suggest three primary culprits:
- Redundancy: Models trained on similar data often share the same blind spots. Axel Abels notes that LLMs are often too similar to one another, meaning simple averaging can actually amplify shared biases rather than mitigating them Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds.
- Noise: In complex reasoning tasks, a large, unmanaged collection of models can produce conflicting outputs that drown out the correct reasoning of a few experts SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models.
- Cognitive Load: In multi-agent debates, Tianyu Hu found that larger groups suffer from the "cognitive load" of processing long-context debate histories, with returns diminishing after roughly seven participants Multi-Agent Debate for LLM Judges with Adaptive Stability Detection.
The Memory Split Advantage
Despite these plateaus, ensembling remains more efficient than building a single "mega-model." A key theoretical insight is the Memory Split Advantage (MSA). Research from NeurIPS 2020 demonstrates that for a fixed parameter budget, you are often better off training an ensemble of several medium-sized networks rather than one massive network.
The intuition is that a single wide network eventually saturates in its ability to generalize. By splitting that capacity into independent models, you leverage variance reduction to push past the performance ceiling of a lone expert Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?.
From Voting to Routing
If the "full ensemble" is often suboptimal, the challenge becomes identifying which models should handle a specific query. This has led to the development of Query-Aware Selection.
Kaushal Kumar Maurya and KV Aditya Srivatsa introduced SelectLLM, which uses a lightweight classifier to route queries to a specialized subset of models. This approach slashes latency by 70% while improving accuracy by avoiding the "noise" of irrelevant models SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models. Similarly, James Kotary proposed e2e-CEL, which treats model selection as a differentiable "Knapsack" problem, learning to pick a "Goldilocks" committee for every input Differentiable Model Selection for Ensemble Learning.
Beyond Simple Aggregation
The most recent frontier in this area moves away from simple voting or averaging toward Inference-Time Architectures.
Instead of treating models as independent voters, frameworks like Archon, developed by Jon Saad-Falcon and colleagues, use Bayesian optimization to design complex interaction patterns. These patterns include cross-model critiquing, multi-layer fusion, and ranking Archon: An Architecture Search Framework for Inference-Time Techniques. By allowing models to "talk" to each other or review each other's work—as seen in the DEI framework for software engineering Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents—researchers can reach performance levels that raw compute scaling alone cannot achieve.
In summary, the field is shifting from quantity to composition. The goal is no longer to build the largest ensemble possible, but to find the smallest, most diverse committee of experts and the right "social choice" mechanism—such as the electoral systems proposed in GEDI An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making—to aggregate their wisdom effectively.