Unveiling Concepts Learned by a World-Class Chess-Playing Agent

Aðalsteinn Pálsson, Yngvi BjörnssonIJCAI 2023 · DOI

TL;DR

This paper explores the "black box" nature of modern, superhuman chess engines like Stockfish, which use neural networks (NNs) for evaluation. By applying interpretability techniques, the authors successfully identify and explain high-level chess concepts learned by the NN, revealing how it differs from traditional, hand-crafted evaluation methods and highlighting its ability to statically recognize complex threats.

What Problem Does This Paper Address?

Modern AI agents for complex games like chess have achieved superhuman performance, largely by replacing intricate, hand-crafted evaluation functions with deep neural networks (NNs). While this has boosted playing strength, it has also made these agents opaque. Understanding why an agent prefers one position over another, or what specific chess knowledge it has internalized, has become a significant challenge. This lack of interpretability hinders our ability to learn from these AIs, build trust in their decisions, and potentially improve human understanding of the game itself. This paper aims to bridge this gap by applying interpretability techniques to a state-of-the-art chess engine to uncover the concepts its neural network has learned.

Methods

The researchers employed two primary categories of interpretability techniques:

  • Model-Agnostic Interpretation: This approach treats the NN as a black box and focuses on understanding the relationship between its inputs and outputs. They used Shapley values to estimate the importance of various chess concepts (like material, king safety, etc.) in the NN's evaluation, comparing these to the weights in Stockfish's traditional, hand-crafted evaluation function. They also used linear surrogate models to explain the NN's evaluation based on these concepts.
  • Model-Specific Interpretation (Concept Probing): This method leverages the internal architecture of the NN. They used concept probing to determine how well specific human-understandable chess concepts are represented in the network's internal activations at different layers. This involves training simple "probes" (classifiers) to predict the presence of a concept from the network's intermediate representations.

Datasets

The primary dataset used for experiments was a collection of 100,000 chess positions randomly sampled from a quality dataset generated by Leela Chess Zero. This dataset was used to compute concepts and static evaluations for both the classical and NNUE models.

Key Insights and Ideas

  • NNs Learn Human-Like Concepts, But Differently: The NNUE model learns many of the same high-level concepts as the classical evaluation (e.g., material, passed pawns), but it assigns different relative importance to them. Notably, it seems to de-emphasize material more than the classical model and places greater importance on dynamic concepts.
  • King Safety is Understood, But Not Explicitly: While the classical model has a dedicated "king safety" feature, the NNUE model's evaluation of king safety is not directly correlated with this feature. However, the NNUE's overall evaluation suggests it has a sophisticated understanding of king safety, likely through a more nuanced combination of other learned features.
  • Static Threat Detection is a Key NNUE Strength: The NNUE model demonstrates an impressive ability to statically recognize threats like forks and potential pawn promotions, which would typically require a look-ahead search in the classical engine. This suggests the NNUE has learned to "see" tactical opportunities directly from the board state.
  • Concept Representation Evolves: Concept probing revealed that concepts become more accurately represented in the NN's hidden layers compared to the raw input, indicating the network is actively learning and distilling these concepts.

Key Results

  • Linear surrogate models explained the classical evaluation with very high fidelity (r² = 0.999), but explained the NNUE evaluation with significantly lower fidelity (r² = 0.497), indicating substantial differences in how the two models evaluate positions.
  • Shapley value analysis showed that while "Material" is the most critical concept for both models, the NNUE places more relative importance on "Winnable" and "Passed Pawns" compared to the classical model.
  • Concept probing showed that concepts like "Winnable" and "Passed Pawns" become more relevant as the game progresses towards the endgame.
  • The NNUE model was observed to statically detect tactical threats (like forks and pawn promotions) that the classical model would miss without search.

Limitations and Future Work

  • Model Specificity: Applying model-specific techniques like concept probing to the NNUE architecture requires careful consideration due to its complex, phase-dependent structure (multiple layer stacks).
  • Concept Vocabulary: The study relied on a predefined set of human-understandable concepts. Future work could explore discovering entirely new concepts learned by the NN.
  • Game Independence: The interpretability methods used are game-independent. The authors suggest applying these techniques to agents trained for general game-playing to identify cross-game concepts.
  • Deeper Analysis: Further investigation into specific concepts like king safety is warranted to fully understand the NNUE's nuanced evaluation.

Summary updated

BibTeX
@inproceedings{palsson2023unveiling,
  author = {Aðalsteinn Pálsson and Yngvi Björnsson},
  title = {{Unveiling Concepts Learned by a World-Class Chess-Playing Agent}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/541},
}
Figure