WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang QiarXiv 2026 · arXiv

TL;DR

WorldCup Arena introduces a "leakage-free" evaluation for Large Language Models (LLMs) by testing their forecasting abilities on a live, future event: the 2026 FIFA World Cup. By forcing models to predict outcomes before kickoff, the researchers prove that current frontier LLMs largely mimic betting market favorites and suffer from a shared "conservatism" that prevents them from outperforming simple baselines.

What Problem Does This Paper Address?

Most LLM benchmarks are retrospective—they test models on data that already exists on the web. This creates a massive risk of data leakage, where a model "predicts" an outcome it has actually already seen in its training data. Even benchmarks that refresh frequently struggle to prove they are truly contamination-free.

The authors argue that forecasting is the "most honest test" of a reasoning system. If an event hasn't happened yet, the answer exists nowhere for the model to recall. To test this, they used the 2026 FIFA World Cup as a dense, 39-day experiment to see if frontier LLMs can actually reason about the future or if they are just sophisticated parrots of existing web sentiment.

Methods

The researchers established a rigorous "prospective" pipeline for six frontier models (including versions of Claude, GPT, Gemini, and others).

  1. The Task: Before every one of the 104 matches, models were given a "dossier" containing team stats, history, and live news. They had to fill out a seven-market prediction card (e.g., match winner, correct score, over/under 2.5 goals) in a structured JSON format.
  2. Three Procedural Locks:
    • Snapshot Lock: Dossiers only contained information available up to that moment.
  3. Kickoff Lock: All predictions were timestamped and locked before the actual match started.
  4. Template Lock: Prompts and scoring weights were fixed in advance to prevent "prompt engineering" the results after the fact.
  5. Model Strength: Models were run at "full strength," utilizing extended thinking capabilities and native server-side web search to mimic a human forecaster.
The prospective evaluation pipeline vs. traditional retrospective studies.

Datasets

The paper introduces the WorldCup Arena Benchmark, which includes:

  • 48 Team Dossiers: 46 timestamped snapshots of team data, growing from 623k to 1.17M characters as the tournament progressed.
  • 104 Fixture Metadata: Venue, weather, referee, and bookmaker handicap lines.
  • 4,494 Scored Predictions: A frozen archive of how the six frontier models performed.

Key Insights and Ideas

  • The "Herding" Effect: The models agree with each other far more often than they are right. As seen in Figure 2, their cumulative scores accrue almost in lockstep, suggesting they share a common reasoning "recipe" that leads to identical mistakes.
  • Market Mimicry: Models don't beat the betting markets; they track them. They sided with the bookmaker's favorite in 85–90% of matches.
  • Shared Conservatism: The models exhibit "draw blindness" and "goal shyness." They under-predicted draws (predicting ~10 when 27 occurred) and crowded their scoreline picks onto "prototypical" results like 2–1.
  • Complexity Collapse: Counter-intuitively, model accuracy was highest in lopsided matches and collapsed in close ties—precisely where the dossiers were the richest and reasoning was most required.
Models gain and lose points in lockstep, showing high agreement (herding).

Key Results

  • Leaderboard: Claude finished first with 897 points, while GLM finished last with 813. However, the margins remained narrow throughout the tournament.
  • Accuracy vs. Baselines: The models averaged 63.9% accuracy on match outcomes (1X2). This is statistically indistinguishable from the 64.4% achieved by simply backing the bookmaker's favorite every time.
  • Market Performance: While models easily beat uniform guessing, they barely cleared the "coin flip" threshold on the high-weighted handicap market.
Per-market accuracy compared to uniform guessing and bookmaker favorites.

Limitations and Future Work

The authors acknowledge that this study is limited to a single domain (football) and a single event. Because the tournament only happens once, the sample size for "tournament-level" questions is small.

Furthermore, the models were forced to make a single choice per market rather than providing a probability distribution. Future work could involve "wisdom of the crowd" experiments with LLM ensembles or expanding the prospective evaluation to other volatile domains like financial markets or political elections to see if the "herding" and "conservatism" behaviors persist.

Summary updated

Abstract

Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.
BibTeX
@misc{wang2026worldcup,
  author = {Zhenran Wang and Zhonghan Bian and Jinsong Li and Zhangyang Qi},
  title = {{WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament}},
  year = {2026},
  eprint = {2608.04008},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.04008},
}
Figure