TL;DR
WorldCup Arena introduces a "leakage-free" evaluation for Large Language Models (LLMs) by testing their forecasting abilities on a live, future event: the 2026 FIFA World Cup. By forcing models to predict outcomes before kickoff, the researchers prove that current frontier LLMs largely mimic betting market favorites and suffer from a shared "conservatism" that prevents them from outperforming simple baselines.
What Problem Does This Paper Address?
Most LLM benchmarks are retrospective—they test models on data that already exists on the web. This creates a massive risk of data leakage, where a model "predicts" an outcome it has actually already seen in its training data. Even benchmarks that refresh frequently struggle to prove they are truly contamination-free.
The authors argue that forecasting is the "most honest test" of a reasoning system. If an event hasn't happened yet, the answer exists nowhere for the model to recall. To test this, they used the 2026 FIFA World Cup as a dense, 39-day experiment to see if frontier LLMs can actually reason about the future or if they are just sophisticated parrots of existing web sentiment.
Methods
The researchers established a rigorous "prospective" pipeline for six frontier models (including versions of Claude, GPT, Gemini, and others).
- The Task: Before every one of the 104 matches, models were given a "dossier" containing team stats, history, and live news. They had to fill out a seven-market prediction card (e.g., match winner, correct score, over/under 2.5 goals) in a structured JSON format.
- Three Procedural Locks:
- Snapshot Lock: Dossiers only contained information available up to that moment.
- Kickoff Lock: All predictions were timestamped and locked before the actual match started.
- Template Lock: Prompts and scoring weights were fixed in advance to prevent "prompt engineering" the results after the fact.
- Model Strength: Models were run at "full strength," utilizing extended thinking capabilities and native server-side web search to mimic a human forecaster.
Datasets
The paper introduces the WorldCup Arena Benchmark, which includes:
- 48 Team Dossiers: 46 timestamped snapshots of team data, growing from 623k to 1.17M characters as the tournament progressed.
- 104 Fixture Metadata: Venue, weather, referee, and bookmaker handicap lines.
- 4,494 Scored Predictions: A frozen archive of how the six frontier models performed.
Key Insights and Ideas
- The "Herding" Effect: The models agree with each other far more often than they are right. As seen in Figure 2, their cumulative scores accrue almost in lockstep, suggesting they share a common reasoning "recipe" that leads to identical mistakes.
- Market Mimicry: Models don't beat the betting markets; they track them. They sided with the bookmaker's favorite in 85–90% of matches.
- Shared Conservatism: The models exhibit "draw blindness" and "goal shyness." They under-predicted draws (predicting ~10 when 27 occurred) and crowded their scoreline picks onto "prototypical" results like 2–1.
- Complexity Collapse: Counter-intuitively, model accuracy was highest in lopsided matches and collapsed in close ties—precisely where the dossiers were the richest and reasoning was most required.
Key Results
- Leaderboard: Claude finished first with 897 points, while GLM finished last with 813. However, the margins remained narrow throughout the tournament.
- Accuracy vs. Baselines: The models averaged 63.9% accuracy on match outcomes (1X2). This is statistically indistinguishable from the 64.4% achieved by simply backing the bookmaker's favorite every time.
- Market Performance: While models easily beat uniform guessing, they barely cleared the "coin flip" threshold on the high-weighted handicap market.
Limitations and Future Work
The authors acknowledge that this study is limited to a single domain (football) and a single event. Because the tournament only happens once, the sample size for "tournament-level" questions is small.
Furthermore, the models were forced to make a single choice per market rather than providing a probability distribution. Future work could involve "wisdom of the crowd" experiments with LLM ensembles or expanding the prospective evaluation to other volatile domains like financial markets or political elections to see if the "herding" and "conservatism" behaviors persist.
Summary updated