Oracle-Guided Soft Shielding for Safe Move Prediction in Chess

Prajit T. Rajendran, Fabio Arnez, Huáscar Espinoza, Agnès Delaborde, Chokri Mraidha

TL;DR

This paper introduces Oracle-Guided Soft Shielding (OGSS), a novel framework for safer decision-making in complex environments like chess. OGSS combines imitation learning with a learned "blunder predictor" that uses expert feedback to identify and avoid risky moves, enabling agents to explore more broadly without sacrificing tactical soundness.

What Problem Does This Paper Address?

Developing intelligent agents that can operate safely in high-stakes environments is a significant challenge. Traditional reinforcement learning (RL) often requires vast amounts of data and computational resources, while imitation learning (IL), though more sample-efficient, can be brittle and prone to catastrophic errors, especially in complex domains like chess. A single tactical blunder can irrevocably alter the game's outcome. Existing methods for ensuring safety often rely on hard-coded rules or continuous human supervision, which are not scalable or practical for real-time deployment. This paper addresses the need for a more flexible, data-driven, and scalable approach to safe exploration that can proactively avoid critical mistakes without sacrificing performance or the ability to learn.

Methods

The core of the proposed solution, Oracle-Guided Soft Shielding (OGSS), consists of two main components:

  1. Move Predictor: This is a standard supervised learning model, trained using imitation learning on historical chess games. Its goal is to predict the most likely expert move given a board state. This component learns the "what" – what a good move looks like.
  2. Blunder Predictor: This is the novel "safety shield." It's trained using feedback from an oracle (in this case, the Stockfish chess engine) to predict the probability that a given move will result in a tactical blunder. A blunder is defined as a move causing a significant drop in the engine's evaluation (over 100 centipawns). This component learns the "what not" – what a risky move looks like.

At inference time, OGSS combines these two components. The agent generates candidate moves using the move predictor and then uses the blunder predictor to assess the risk associated with each. The paper proposes three variants for selecting the final move:

  • Action Elimination: Moves exceeding a predefined risk threshold are eliminated. If no safe move exists, the highest-confidence move is chosen.
  • Utility-Based Selection: A utility function balances the move's confidence score with its predicted risk probability, allowing for a tunable trade-off between performance and safety.
  • Top-K Shielding: The agent first considers the top-K most confident moves and then selects the one with the lowest predicted blunder probability.

This approach decouples performance learning from risk modeling, allowing for a modular and flexible safety mechanism.

Datasets

The paper uses two primary sources of data:

  • Historical Chess Games: 10,000 decisive (checkmate) games from the Lichess dataset were used to train the initial move predictor via imitation learning.
  • Stockfish Engine Feedback: The Stockfish chess engine was used as an oracle to generate labels for the blunder predictor. Moves that caused a significant drop in evaluation were labeled as blunders.

The evaluation was conducted by playing games against the Stockfish chess engine itself.

Key Insights and Ideas

  • Learned Safety is Scalable: The paper demonstrates that a probabilistic safety filter can be learned from data (oracle feedback) rather than relying on brittle, hand-crafted rules. This makes the approach scalable to complex domains like chess.
  • Soft Shielding for Flexibility: Unlike hard safety filters that simply block actions, OGSS uses a probabilistic approach. This allows the agent to make nuanced trade-offs between performance and safety, enabling exploration without being overly conservative.
  • Proactive Risk Avoidance: The blunder predictor acts proactively, identifying potentially dangerous moves before they are executed, rather than reactively correcting errors.
  • Balancing Exploration and Safety: The key finding is that OGSS variants can achieve significantly lower blunder rates while maintaining higher exploration ratios compared to baselines like SafeDAgger and action pruning. This suggests that safe exploration doesn't necessarily mean sacrificing the ability to discover new strategies.
  • Oracle Feedback as a Powerful Tool: The effectiveness of Stockfish's tactical evaluations as a source for training a safety model highlights the value of expert feedback in learning safe behaviors.

Key Results

  • Lowest Blunder Rate: OGSS (action elimination) achieved the lowest blunder rate (0.2411 ± 0.0097), slightly outperforming the SafeDAgger + greedy baseline.
  • Higher Exploration: OGSS variants, particularly OGSS (top-5 + blunder shield), achieved higher exploration ratios (up to 0.4091 ± 0.0261) while maintaining low blunder rates, demonstrating the ability to explore more broadly without increased risk.
  • Competitive Move Quality: OGSS (action elimination) also achieved the lowest median centipawn drop (24.42 ± 2.77), indicating that reducing blunders did not significantly compromise the quality of non-blunder moves.
  • Robustness: The method showed good generalization, even in data-scarce settings.

Limitations and Future Work

  • Oracle Dependency: The current framework relies on an oracle (Stockfish) for training the blunder predictor. While this oracle is used offline, the scalability of this approach to domains without readily available, high-quality oracles remains an open question.
  • Exploration vs. Exploitation Trade-off: While OGSS offers a better balance, the paper acknowledges the inherent trade-off between exploration and exploitation. The alpha parameter in the utility-based variant provides a tunable knob for this, but finding optimal settings might require further investigation.
  • Future Work: The authors suggest that the core architecture is modality-agnostic and could be applied to other domains where expert feedback on risky behavior is available. Further exploration into optimizing the exploration-safety trade-off and reducing reliance on expensive oracles are potential future directions.

Summary updated

BibTeX
@inproceedings{rajendran2025oracle,
  author = {Prajit T. Rajendran and Fabio Arnez and Huáscar Espinoza and Agnès Delaborde and Chokri Mraidha},
  title = {{Oracle-Guided Soft Shielding for Safe Move Prediction in Chess}},
  booktitle = {ICMLA},
  year = {2025},
  doi = {10.1109/icmla66185.2025.00130},
}
Figure