Abstract
The AI landscape is currently dominated by massive proprietary models that require immense resources to maintain. This paper introduces The Avengers, a training-free framework that orchestrates a collective of small, open-source language models (SLMs) to rival and exceed the performance of giants like GPT-4. By employing semantic clustering for query routing and majority voting for output refinement, the system demonstrates that a "team" of specialized ~7B-parameter models can outperform massive closed-source alternatives across diverse benchmarks, achieving an average score of 70.54 compared to GPT-4.1's 69.20.
Introduction
The rapid advancement of Large Language Models (LLMs) has led to a "closed competition" dominated by proprietary giants like GPT-4o. While these models offer broad general-purpose capabilities, they marginalize the open-source community. Conversely, smaller, open-source models (SLMs) often excel in specific niches but struggle to match the comprehensive performance of their larger counterparts.
The authors address this gap by asking whether a team of smaller models, orchestrated effectively, can remain competitive across a wide range of tasks. The challenge lies in "collective intelligence": selecting the right subset of models and routing queries to the most capable experts without the overhead of training complex neural routers.
Methodology: The Avengers Framework
The proposed framework, The Avengers, operates as a training-free "simple recipe" divided into two distinct stages: offline calibration and online inference.
Stage I: Offline Calibration
In this stage, the system builds a "capability profile" for each candidate model. Queries from a validation set are converted into semantic embeddings and grouped into $K$ clusters (e.g., $K=64$) using standard algorithms like K-Means. Every model is evaluated on these clusters to identify its strengths—for instance, identifying which models excel at "Python coding" versus "medical advice." Finally, the system automatically selects a diverse "team" of models that provides the best collective coverage across all semantic neighborhoods.
Stage II: Online Inference
When a new query arrives, it is embedded and assigned to the nearest semantic cluster identified during calibration. The system routes the query to the top-performing model(s) for that specific cluster. To further refine the output, the selected model generates multiple responses through repeated sampling, and a final answer is determined via majority voting (Self-Consistency).
Experimental Evaluation
The framework was evaluated across 15 diverse datasets spanning five categories: Mathematics (AIME, Math500, LiveMathBench), Code (MBPP, HumanEval), Logic (KORBench, BBH), Knowledge (MMLUPro, MedQA), and Affective/Emotion (MELD). To test robustness, the researchers also utilized five Out-of-Distribution (OOD) datasets.
Key Results and Insights
The Avengers framework demonstrated that diversity often beats raw power. By selecting specialized models—some of which have low community visibility but high niche expertise—the system achieved superior results to general-purpose giants.
- Surpassing Proprietary Models: Using 10 models of approximately 7B parameters each, The Avengers achieved an average score of 70.54, surpassing GPT-4.1 (69.20).
- Domain Specialization: The system showed significant dominance in technical fields, outperforming GPT-4.1 by 18.21% in Mathematics and 7.46% in Coding benchmarks.
- Robustness of Simple Clustering: The authors found that simple semantic clustering is more robust than complex neural routers, which often overfit on training data. The Avengers remained stable across a wide range of cluster counts ($K=14$ to $140$).
- Training-Free Advantage: As a "plug-and-play" system, new models can be added to the team simply by updating capability profiles through validation set evaluation, avoiding the need for backpropagation or fine-tuning.
Conclusion and Future Work
The Avengers proves that collective intelligence among smaller models is a viable path toward challenging the dominance of proprietary AI. While the system currently requires a labeled validation set for calibration and has primarily been tested on objective tasks (math, code, multiple-choice), it opens new doors for the open-source community. Future research will explore scaling this recipe to larger model sizes (e.g., 70B) and validating its performance on subjective, conversational tasks.
Summary updated