The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants

Yiqun Zhang, Hao Li, Chenxu Wang, Linyao Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Daling Wang, Zhen Wang, Xinrun Wang, Xu Jia, Lei Bai, Wanli Ouyang, Shuyue HuarXiv 2025 · DOI · arXiv

Abstract

The AI landscape is currently dominated by massive proprietary models that require immense resources to maintain. This paper introduces The Avengers, a training-free framework that orchestrates a collective of small, open-source language models (SLMs) to rival and exceed the performance of giants like GPT-4. By employing semantic clustering for query routing and majority voting for output refinement, the system demonstrates that a "team" of specialized ~7B-parameter models can outperform massive closed-source alternatives across diverse benchmarks, achieving an average score of 70.54 compared to GPT-4.1's 69.20.

Introduction

The rapid advancement of Large Language Models (LLMs) has led to a "closed competition" dominated by proprietary giants like GPT-4o. While these models offer broad general-purpose capabilities, they marginalize the open-source community. Conversely, smaller, open-source models (SLMs) often excel in specific niches but struggle to match the comprehensive performance of their larger counterparts.

The authors address this gap by asking whether a team of smaller models, orchestrated effectively, can remain competitive across a wide range of tasks. The challenge lies in "collective intelligence": selecting the right subset of models and routing queries to the most capable experts without the overhead of training complex neural routers.

Methodology: The Avengers Framework

The proposed framework, The Avengers, operates as a training-free "simple recipe" divided into two distinct stages: offline calibration and online inference.

The two-stage framework of The Avengers: Offline Calibration for model profiling and Online Inference for query routing and voting.

Stage I: Offline Calibration

In this stage, the system builds a "capability profile" for each candidate model. Queries from a validation set are converted into semantic embeddings and grouped into $K$ clusters (e.g., $K=64$) using standard algorithms like K-Means. Every model is evaluated on these clusters to identify its strengths—for instance, identifying which models excel at "Python coding" versus "medical advice." Finally, the system automatically selects a diverse "team" of models that provides the best collective coverage across all semantic neighborhoods.

Stage II: Online Inference

When a new query arrives, it is embedded and assigned to the nearest semantic cluster identified during calibration. The system routes the query to the top-performing model(s) for that specific cluster. To further refine the output, the selected model generates multiple responses through repeated sampling, and a final answer is determined via majority voting (Self-Consistency).

Experimental Evaluation

The framework was evaluated across 15 diverse datasets spanning five categories: Mathematics (AIME, Math500, LiveMathBench), Code (MBPP, HumanEval), Logic (KORBench, BBH), Knowledge (MMLUPro, MedQA), and Affective/Emotion (MELD). To test robustness, the researchers also utilized five Out-of-Distribution (OOD) datasets.

Key Results and Insights

The Avengers framework demonstrated that diversity often beats raw power. By selecting specialized models—some of which have low community visibility but high niche expertise—the system achieved superior results to general-purpose giants.

Performance comparison of The Avengers against GPT-4o, GPT-4.5-Preview, and GPT-4.1 across major benchmarks.
  • Surpassing Proprietary Models: Using 10 models of approximately 7B parameters each, The Avengers achieved an average score of 70.54, surpassing GPT-4.1 (69.20).
  • Domain Specialization: The system showed significant dominance in technical fields, outperforming GPT-4.1 by 18.21% in Mathematics and 7.46% in Coding benchmarks.
  • Robustness of Simple Clustering: The authors found that simple semantic clustering is more robust than complex neural routers, which often overfit on training data. The Avengers remained stable across a wide range of cluster counts ($K=14$ to $140$).
  • Training-Free Advantage: As a "plug-and-play" system, new models can be added to the team simply by updating capability profiles through validation set evaluation, avoiding the need for backpropagation or fine-tuning.

Conclusion and Future Work

The Avengers proves that collective intelligence among smaller models is a viable path toward challenging the dominance of proprietary AI. While the system currently requires a labeled validation set for calibration and has primarily been tested on objective tasks (math, code, multiple-choice), it opens new doors for the open-source community. Future research will explore scaling this recipe to larger model sizes (e.g., 70B) and validating its performance on subjective, conversational tasks.

Summary updated

Abstract

Proprietary giants are increasingly dominating the race for ever-larger language models. Can open-source, smaller models remain competitive across a broad range of tasks? In this paper, we present the Avengers -- a simple recipe that leverages the collective intelligence of these smaller models. The Avengers builds upon four lightweight operations: (i) embedding: encode queries using a text embedding model; (ii) clustering: group queries based on their semantic similarity; (iii) scoring: scores each model's performance within each cluster; and (iv) voting: improve outputs via repeated sampling and voting. At inference time, each query is embedded and assigned to its nearest cluster. The top-performing model(s) within that cluster are selected to generate the response with repeated sampling. Remarkably, with 10 open-source models (~7B parameters each), the Avengers surpasses GPT-4o, 4.1, and 4.5 in average performance across 15 diverse datasets spanning mathematics, coding, logical reasoning, general knowledge, and affective tasks. In particular, it surpasses GPT-4.1 on mathematics tasks by 18.21% and on code tasks by 7.46%. Furthermore, the Avengers delivers superior out-of-distribution generalization, and remains robust across various embedding models, clustering algorithms, ensemble strategies, and values of its sole parameter -- the number of clusters.
BibTeX
@misc{zhang2025avengers,
  author = {Yiqun Zhang and Hao Li and Chenxu Wang and Linyao Chen and Qiaosheng Zhang and Peng Ye and Shi Feng and Daling Wang and Zhen Wang and Xinrun Wang and Xu Jia and Lei Bai and Wanli Ouyang and Shuyue Hu},
  title = {{The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants}},
  year = {2025},
  eprint = {2505.19797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19797},
}
Figure