LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess

Sai Kolasani, Maxim Saplin, Nicholas Crispino, Kyle Montgomery, Jared Quincy Davis, Matei Zaharia, Chi Wang, Chenguang WangSubmission to ICLR 2026 · Rejected

TL;DR

LLM Chess is a new interactive benchmark that tests both reasoning and multi-turn instruction-following in large language models through real-time chess. Even when provided with legal move retrieval tools, standard models almost always forfeit against a completely random opponent due to tool-use failures. Top reasoning models fare better against random play, yet the best model tested (OpenAI’s o3) achieved an estimated Elo of only ~758 against a calibrated chess engine—barely exceeding an average casual human player.

What Problem Does This Paper Address?

While modern LLMs reach expert or competition-grade performance on static benchmarks in math (AIME) and code (Codeforces, LiveCodeBench), these evaluation sets suffer from rapid saturation, memorization, and test-set contamination.

Games like chess demand long-horizon planning, precise state tracking, and adaptive decision-making. However, previous evaluations often tested chess via single-turn next-move prediction or static PGN completions. The authors ask: Can LLMs act as autonomous chess agents across extended, multi-turn interactions, and does their advertised reasoning prowess translate to game-playing depth?

Methods

The authors introduce LLM Chess, built using the AG2 multi-agent framework:

  • Agentic Interaction Loop: At each ply (half-move), an LLM playing as Black engages in a turn-based dialogue with an environment proxy. It has access to three discrete actions:
  • get_current_board (returns the board in Unicode)
  • get_legal_moves (returns valid moves in UCI format)
  • make_move <UCI> (executes the move and ends the turn)
  • Constraint Boundaries: Games are capped at 100 moves (200 plys). The LLM is granted up to 10 conversation turns per ply and 3 attempts per turn to output a valid action. To mirror chess engine state-evaluation constraints, move history is omitted from the default prompt.
  • Two-Phase Evaluation:
  • Gating Phase: 30 games against a Random Agent (which picks randomly among legal moves) to measure baseline instruction-following and completion rates.
  • Engine Phase: Promising models face Komodo’s Dragon 1 engine (calibrated to Chess.com ratings across skills 1–10). Game outcomes yield a maximum-likelihood Elo rating with a 35-point adjustment for playing as Black.
  • Move Quality Metrics: Moves are analyzed via Stockfish 17 (depth 20) to quantify Centipawn Loss, Win% swings, Blunders ($\Delta \ge 30\%$), Mistakes, and Inaccuracies.

Datasets

The paper does not rely on static text datasets:

  • LLM Chess Benchmark & Leaderboard: Evaluates over 50 open and closed-source models across thousands of agentic game traces.
  • External Comparison: Models are cross-referenced with Pass@1 performance on LiveCodeBench.
  • All game logs, environment code, and the public leaderboard are released open-source.

Key Insights and Ideas

  • A Stark Reasoning Divide: Non-reasoning models fail overwhelmingly. Across 29 standard models, the average win rate against a random opponent was just 0.7%, with 71.9% of games lost directly due to instruction-following failures (e.g., repeating invalid actions or looping). Reasoning-enhanced models cut these instruction failures to 24.4% and averaged a 45.4% win rate.
  • Agentic Tool Fragility: Ablation experiments revealed that reasoning models struggle with the tool-calling setup itself. When get_current_board and get_legal_moves were removed and provided directly in the prompt (leaving only make_move), o4-mini (low)'s win rate against random play jumped from 73.3% to 96.7%.
  • Reasoning Depth Beats Ensemble Width: Scaling reasoning compute (low $\to$ medium $\to$ high) boosted win rates by up to 20%. Conversely, Mixture-of-Agents (MoA) parallel proposers yielded diminishing returns. However, pairing a strong reasoner with poor instruction-following (e.g., DeepSeek-R1) with a strict instruction-following synthesizer (GPT-4.1-mini) rescued completion rates to 100%.
  • The "Drosophila" Reality Check: Performance on LLM Chess correlates moderately with LiveCodeBench ($r = 0.686$), confirming that chess exercises general reasoning. Yet despite reaching the 90th percentile in competitive programming, the best-tested LLM peaked at 758 Elo—far below human club players or specialized engines.

Key Results

  • Versus Random Opponents:
  • 35 models achieved a 0% Win/Loss, failing entirely due to parsing errors or conversational deadlocks.
  • Only o3 (medium) scored a perfect 100% win rate.
  • Stockfish analysis showed o4-mini (medium) made only 4.2% blunders and found the engine's best move 19.5% of the time, compared to 31.3% blunders and 4.1% best moves for GPT-4.1-mini.
  • Calibrated Elo vs. Dragon 1:
  • o3 (low): $758 \pm 65$ Elo (managed a 71.9% win/loss against Skill 5, but crashed to 3.0% at Skill 10).
  • Grok 3 Mini (high): $456 \pm 59$ Elo.
  • o3-mini (high): $438 \pm 95$ Elo.
  • o4-mini (high): $408 \pm 100$ Elo.
  • o3-mini (low): $-85 \pm 192$ Elo (baseline Random Agent measured at $-122.3$).

Limitations and Future Work

  • Absence of Move History: While aligning with standard engine evaluation, withholding move history deviates from human play; adding past moves lowered blunder rates from 11.2% to 1.6% for o4-mini (low).
  • Fixed Role (Black): Models were evaluated almost exclusively as Black, requiring a statistical Elo offset (+35) rather than balanced two-sided play.
  • High-Compute API Timeouts: Higher reasoning tiers of OpenAI models frequently hit the 10-minute AG2 client timeout, registering technical forfeits.
  • Future Scope: Extending the framework to dynamically scale engine skill levels as LLM capabilities progress, preventing ceiling effects without relying on static puzzles.

Summary updated

Abstract

We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation, proving difficult even for top reasoning models. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.
BibTeX
@inproceedings{kolasani2025llm,
  author = {Sai Kolasani and Maxim Saplin and Nicholas Crispino and Kyle Montgomery and Jared Quincy Davis and Matei Zaharia and Chi Wang and Chenguang Wang},
  title = {{LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess}},
  booktitle = {FoRLM @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=i0UHrCKEXZ},
}
Figure