Emergent evaluation hubs in a decentralizing large language model ecosystem

Manuel Cebrian, Tomomi Kito, Raul Castro FernandezarXiv 2025 · DOI · arXiv

Abstract

As the production of Large Language Models (LLMs) rapidly decentralizes across a growing number of countries and organizations, the benchmarks used to evaluate them are experiencing the opposite trend. This paper investigates the tension between a multipolar world of model production and a unipolar landscape of evaluation. Through network science and agent-based modeling, we reveal that evaluation influence is concentrating into a few elite "hubs," creating a high-stakes trade-off between global standardization and a dangerously narrow evaluative lens.

The Problem: A Bottleneck in Evaluation

The LLM ecosystem is currently undergoing an explosion in diversity, with hundreds of models emerging from varied geographic and organizational origins. To ensure this growth is meaningful, the community requires common benchmarks to compare performance, safety, and impact. However, a critical question remains: Is the infrastructure for evaluating these models evolving alongside the models themselves, or is it becoming a bottleneck? This study explores whether evaluative attention is diffuse or concentrated and identifies the mechanisms, such as the "Matthew Effect" of cumulative advantage, driving these patterns.

The Great Divergence: Expansion vs. Concentration

Our analysis identifies a phenomenon termed "The Great Divergence." While model production is decentralizing—with more players and countries entering the field—benchmark influence is centralizing. We are increasingly seeing a world where a vast array of models are judged by a very small, dominant set of tests.

Trends in model releases, parameter counts, and manufacturer diversity showing rapid ecosystem expansion.

Statistical metrics confirm this extreme concentration. The top 15% of nodes in the benchmark network account for over 80% of high-betweenness paths. Geographically, just three countries produce 83% of benchmark outputs, and the global Gini coefficient for benchmark authority stands at a staggering 0.89. Institutional dominance is equally pronounced, with Google, Stanford, and UC Berkeley collectively holding nearly 50% of measured benchmark authority.

The Transparency Paradox

As the number of models grows, the quality of documentation is paradoxically declining. Our research shows that the explicit reporting of critical metadata—such as training emissions, hardware usage, and runtime—peaked in 2020 and has eroded since. This suggests that the rapid proliferation of models has come at the cost of openness and reproducibility.

The decline in reporting of key model metadata and the shifting landscape of license and weight availability.

Methodology and Mechanisms

To reach these conclusions, we employed a multi-disciplinary approach:

  • Network Analysis: Constructing tripartite graphs of benchmarks, authors, and institutions to map "authority."
  • Statistical Metrics: Utilizing Gini coefficients and the Herfindahl-Hirschman Index (HHI) to quantify inequality.
  • Principal Component Analysis (PCA): Condensing ecosystem indicators into two primary axes: "Expansion" (growth) and "Transparency" (documentation quality).
  • Agent-Based Modeling (ABM): Simulating the ecosystem to understand why concentration occurs.

The simulation confirmed that "preferential attachment" is the dominant force. Popular benchmarks like MMLU or HumanEval attract more citations and stars, leading to even greater adoption. Interestingly, the only factor that significantly reduced concentration in simulations was the entry rate of entirely novel benchmarks; increasing penalties for reusing old, "stale" benchmarks had almost no effect.

Discussion and Future Directions

Concentration in evaluation isn't purely a failure; it functions as a necessary "coordination infrastructure," providing a common language for comparison. However, the current level of centralization risks creating a narrow evaluative lens that may ignore regional, multi-lingual, or culturally specific nuances.

Future work must focus on "benchmark audits" and funding programs that target the development of diverse evaluation suites. Preventing the evaluation landscape from becoming a closed loop is essential for the long-term health and safety of the global AI ecosystem.

Summary updated

BibTeX
@misc{cebrian2025emergent,
  author = {Manuel Cebrian and Tomomi Kito and Raul Castro Fernandez},
  title = {{Emergent evaluation hubs in a decentralizing large language model ecosystem}},
  year = {2025},
  eprint = {2510.01286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01286},
}
Figure