TL;DR
LLM Chess is a new interactive benchmark that tests both reasoning and multi-turn instruction-following in large language models through real-time chess. Even when provided with legal move retrieval tools, standard models almost always forfeit against a completely random opponent due to tool-use failures. Top reasoning models fare better against random play, yet the best model tested (OpenAI’s o3) achieved an estimated Elo of only ~758 against a calibrated chess engine—barely exceeding an average casual human player.
What Problem Does This Paper Address?
While modern LLMs reach expert or competition-grade performance on static benchmarks in math (AIME) and code (Codeforces, LiveCodeBench), these evaluation sets suffer from rapid saturation, memorization, and test-set contamination.
Games like chess demand long-horizon planning, precise state tracking, and adaptive decision-making. However, previous evaluations often tested chess via single-turn next-move prediction or static PGN completions. The authors ask: Can LLMs act as autonomous chess agents across extended, multi-turn interactions, and does their advertised reasoning prowess translate to game-playing depth?
Methods
The authors introduce LLM Chess, built using the AG2 multi-agent framework:
- Agentic Interaction Loop: At each ply (half-move), an LLM playing as Black engages in a turn-based dialogue with an environment proxy. It has access to three discrete actions:
get_current_board(returns the board in Unicode)get_legal_moves(returns valid moves in UCI format)make_move <UCI>(executes the move and ends the turn)- Constraint Boundaries: Games are capped at 100 moves (200 plys). The LLM is granted up to 10 conversation turns per ply and 3 attempts per turn to output a valid action. To mirror chess engine state-evaluation constraints, move history is omitted from the default prompt.
- Two-Phase Evaluation:
- Gating Phase: 30 games against a Random Agent (which picks randomly among legal moves) to measure baseline instruction-following and completion rates.
- Engine Phase: Promising models face Komodo’s Dragon 1 engine (calibrated to Chess.com ratings across skills 1–10). Game outcomes yield a maximum-likelihood Elo rating with a 35-point adjustment for playing as Black.
- Move Quality Metrics: Moves are analyzed via Stockfish 17 (depth 20) to quantify Centipawn Loss, Win% swings, Blunders ($\Delta \ge 30\%$), Mistakes, and Inaccuracies.
Datasets
The paper does not rely on static text datasets:
- LLM Chess Benchmark & Leaderboard: Evaluates over 50 open and closed-source models across thousands of agentic game traces.
- External Comparison: Models are cross-referenced with Pass@1 performance on LiveCodeBench.
- All game logs, environment code, and the public leaderboard are released open-source.
Key Insights and Ideas
- A Stark Reasoning Divide: Non-reasoning models fail overwhelmingly. Across 29 standard models, the average win rate against a random opponent was just 0.7%, with 71.9% of games lost directly due to instruction-following failures (e.g., repeating invalid actions or looping). Reasoning-enhanced models cut these instruction failures to 24.4% and averaged a 45.4% win rate.
- Agentic Tool Fragility: Ablation experiments revealed that reasoning models struggle with the tool-calling setup itself. When
get_current_boardandget_legal_moveswere removed and provided directly in the prompt (leaving onlymake_move),o4-mini (low)'s win rate against random play jumped from 73.3% to 96.7%. - Reasoning Depth Beats Ensemble Width: Scaling reasoning compute (low $\to$ medium $\to$ high) boosted win rates by up to 20%. Conversely, Mixture-of-Agents (MoA) parallel proposers yielded diminishing returns. However, pairing a strong reasoner with poor instruction-following (e.g., DeepSeek-R1) with a strict instruction-following synthesizer (GPT-4.1-mini) rescued completion rates to 100%.
- The "Drosophila" Reality Check: Performance on LLM Chess correlates moderately with LiveCodeBench ($r = 0.686$), confirming that chess exercises general reasoning. Yet despite reaching the 90th percentile in competitive programming, the best-tested LLM peaked at 758 Elo—far below human club players or specialized engines.
Key Results
- Versus Random Opponents:
- 35 models achieved a 0% Win/Loss, failing entirely due to parsing errors or conversational deadlocks.
- Only o3 (medium) scored a perfect 100% win rate.
- Stockfish analysis showed
o4-mini (medium)made only 4.2% blunders and found the engine's best move 19.5% of the time, compared to 31.3% blunders and 4.1% best moves forGPT-4.1-mini. - Calibrated Elo vs. Dragon 1:
- o3 (low): $758 \pm 65$ Elo (managed a 71.9% win/loss against Skill 5, but crashed to 3.0% at Skill 10).
- Grok 3 Mini (high): $456 \pm 59$ Elo.
- o3-mini (high): $438 \pm 95$ Elo.
- o4-mini (high): $408 \pm 100$ Elo.
- o3-mini (low): $-85 \pm 192$ Elo (baseline Random Agent measured at $-122.3$).
Limitations and Future Work
- Absence of Move History: While aligning with standard engine evaluation, withholding move history deviates from human play; adding past moves lowered blunder rates from 11.2% to 1.6% for
o4-mini (low). - Fixed Role (Black): Models were evaluated almost exclusively as Black, requiring a statistical Elo offset (+35) rather than balanced two-sided play.
- High-Compute API Timeouts: Higher reasoning tiers of OpenAI models frequently hit the 10-minute AG2 client timeout, registering technical forfeits.
- Future Scope: Extending the framework to dynamically scale engine skill levels as LLM capabilities progress, preventing ceiling effects without relying on static puzzles.
Summary updated