Learning to Play Chess from Textbooks (LEAP): a Corpus for Evaluating Chess Moves based on Sentiment Analysis

Haifa Alrdahi, Riza Batista-NavarroarXiv 2023 · DOI · arXiv

Introduction

The evolution of chess artificial intelligence has largely followed two paradigms: brute-force search and deep reinforcement learning from structured game data. While engines like AlphaZero have achieved superhuman performance, they operate in a vacuum of "pure calculation," ignoring the rich, qualitative strategic advice found in human literature. Researchers have introduced LEAP (LEArning to Play chess), a novel corpus designed to bridge the gap between natural language descriptions and structured board states. By applying sentiment analysis to classic chess textbooks, this research explores whether AI can learn to evaluate moves by "understanding" the qualitative assessments provided by grandmasters.

The Challenge of Unstructured Chess Knowledge

Mining chess textbooks presents unique linguistic and technical hurdles. Unlike standard move databases (PGN), textbooks contain "unstructured" data where the value of a move is often implied rather than stated. The authors identify four primary challenges:

  1. State Dependency: A move’s quality is entirely dependent on the specific board configuration (FEN) and the preceding sequence.
  2. Implicitness: Grandmasters often describe the consequences of a move without explicitly naming the move itself.
  3. Counterfactual Reasoning: Authors frequently discuss "what-if" scenarios (e.g., "If White had played X, they would have lost"), which can mislead simple models into associating a move with the wrong outcome.
  4. Notation Evolution: Historical texts often use descriptive notation (e.g., "P-K4") which must be translated into modern Standard Algebraic Notation (SAN) for engine compatibility.

Methodology: The LEAP Pipeline

To address these challenges, the researchers developed a pipeline to digitize and annotate Edward Lasker’s 1918 classic, Chess Strategy.

Data Grounding and Annotation

The process began with using regular expressions to parse board diagrams into Forsyth–Edwards Notation (FEN) and converting descriptive notation into SAN. The researchers then performed a two-stage manual annotation of 1,164 sentences:

  • Topic Relevance: Determining if a sentence actually discusses a specific move.
  • Sentiment Analysis: Classifying the author's view of a move as Positive, Negative, Neutral, or Uncertain.

Data Augmentation via DINO

Recognizing that 1,164 sentences constitute a "small" dataset for deep learning, the team employed the DINO method. Utilizing GPT-2-xl, they generated over 80,000 synthetic sentences to help Transformer models generalize across the niche vocabulary of chess.

The LEAP Corpus

The resulting LEAP corpus is a heterogeneous dataset that includes:

  • Structured Data: Move notations and board states (FEN/PGN).
  • Unstructured Data: 1,164 human-annotated sentences from professional literature.
  • Synthetic Data: 82,145 generated data points to bolster training.

Unlike existing datasets, LEAP links evaluative linguistic sentiment directly to specific board states, providing a ground truth for how humans describe strategic advantages.

Key Insights: Sentiment as a Heuristic

The core innovation of LEAP is treating sentiment analysis as a proxy for a chess evaluation function. If a model can detect a grandmaster’s "positive" sentiment toward a maneuver, it can assign that move a higher heuristic value without needing to calculate millions of variations.

The study also revealed a "Positivity Bias" in chess pedagogy. Textbooks contain nearly twice as many positive sentences as negative ones, reflecting a human tendency to teach what to do rather than cataloging every possible error. Furthermore, the researchers noted that sentiment is highly contextual; a phrase like "White wins a piece" is positive for White but negative for Black, requiring the model to track which player the sentiment targets.

Experimental Results

The researchers benchmarked several Transformer models, including BERT, RoBERTa, XLNet, and ALBERT.

  • Relevance Classification: Models performed exceptionally well, identifying move-related sentences with a weighted macro F1 score of up to 97%.
  • Sentiment Analysis: This proved more difficult, with RoBERTa-base achieving a weighted micro F1 score of 68%.

Interestingly, the study found that model size did not strictly correlate with performance. ALBERT-large underperformed compared to smaller baselines, suggesting that for domain-specific tasks like chess, architecture and weight initialization are more critical than raw parameter counts.

Conclusion and Future Work

The LEAP corpus represents a significant step toward "thematic" AI that can explain its reasoning in human terms. While current sentiment analysis is relatively coarse, future work aims to implement Aspect-Based Sentiment Analysis (ABSA) to link sentiment to specific strategic concepts like "king safety" or "pawn structure." Ultimately, the goal is to integrate these sentiment-based evaluations into search engines like Monte Carlo Tree Search (MCTS), allowing AI to play with the strategic intuition of a human grandmaster.

Summary updated

Abstract

Learning chess strategies has been investigated widely, with most studies focussing on learning from previous games using search algorithms. Chess textbooks encapsulate grandmaster knowledge, explain playing strategies and require a smaller search space compared to traditional chess agents. This paper examines chess textbooks as a new knowledge source for enabling machines to learn how to play chess -- a resource that has not been explored previously. We developed the LEAP corpus, a first and new heterogeneous dataset with structured (chess move notations and board states) and unstructured data (textual descriptions) collected from a chess textbook containing 1164 sentences discussing strategic moves from 91 games. We firstly labelled the sentences based on their relevance, i.e., whether they are discussing a move. Each relevant sentence was then labelled according to its sentiment towards the described move. We performed empirical experiments that assess the performance of various transformer-based baseline models for sentiment analysis. Our results demonstrate the feasibility of employing transformer-based sentiment analysis models for evaluating chess moves, with the best performing model obtaining a weighted micro F_1 score of 68%. Finally, we synthesised the LEAP corpus to create a larger dataset, which can be used as a solution to the limited textual resource in the chess domain.
BibTeX
@misc{alrdahi2023learning,
  author = {Haifa Alrdahi and Riza Batista-Navarro},
  title = {{Learning to Play Chess from Textbooks (LEAP): a Corpus for Evaluating Chess Moves based on Sentiment Analysis}},
  year = {2023},
  eprint = {2310.20260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.20260},
}
Figure