Training Characteristic Functions with Reinforcement Learning: XAI-methods play Connect Four

Stephan Wäldchen, Felix Huber, Sebastian Pokutta

TL;DR

This paper introduces a novel approach to evaluating Explainable AI (XAI) methods by training agents to play Connect Four with partially hidden information. This setup allows for direct training of characteristic functions, avoiding the problematic "off-manifold" counterfactuals common in existing XAI evaluation, and enables a more robust comparison of saliency methods through gameplay.

What Problem Does This Paper Address?

Explainable AI (XAI) aims to make complex machine learning models, particularly deep neural networks, more transparent. A key aspect of XAI is "saliency attribution," which identifies which input features were most important for a model's decision. While cooperative game theory offers theoretically sound attribution methods like Shapley values, applying them to black-box models is challenging. Existing methods often rely on creating "counterfactual" inputs that are not representative of real data (off-manifold), making them susceptible to manipulation and difficult to evaluate reliably. The paper addresses this by proposing a new evaluation framework that sidesteps these issues.

Methods

The core innovation is a new setup for training and evaluating XAI methods. Instead of trying to extract characteristic functions from pre-trained black-box models, the authors directly train neural network agents (using Proximal Policy Optimization - PPO) to play Connect Four. Crucially, during training, the agents are exposed to partially hidden information (specifically, the color of pieces on the board). This forces the agents to learn to make decisions based on incomplete data, effectively training them to act as characteristic functions.

For evaluating XAI methods, these trained agents are used as "players." A "masker" (which is an XAI method) determines which features (piece colors) are revealed to the player agent. The player then makes a move based on this partial information. This setup has several advantages:

  • On-manifold evaluation: The partial inputs are directly derived from the training data distribution, avoiding off-manifold issues.
  • Direct characteristic function training: The agents themselves embody the characteristic function.
  • Gameplay-based comparison: XAI methods are compared by their ability to help the agent make good moves in a game.

The paper also explores two specific XAI methods within this framework: Shapley Sampling (a theoretically grounded approach) and a Frank-Wolfe (FW) based method (a heuristic approach).

Datasets

The paper uses the game of Connect Four as its benchmark. The "dataset" is effectively generated through self-play of the trained Connect Four agents. No external datasets are used for training or evaluation.

Key Insights and Ideas

  • Training on partial information is a viable proxy for XAI evaluation: By training agents with deliberately hidden information, we can create a scenario where understanding feature importance directly impacts performance.
  • Avoiding off-manifold counterfactuals is crucial for reliable XAI evaluation: The paper demonstrates how existing evaluation methods can be manipulated by exploiting off-manifold inputs, leading to misleading explanations. Their Connect Four setup inherently avoids this.
  • Heuristic XAI methods can perform competitively with theoretically grounded ones: In their Connect Four tournament, methods like DeepShap and GuidedBackprop (which are heuristic) performed as well as or better than Shapley Sampling, suggesting that practical performance can sometimes outweigh theoretical purity.
  • The "ground truth" for a decision can be defined by the pieces that complete a winning line: This provides a concrete way to assess how well XAI methods identify the most critical features for a specific move.
  • Training objective matters: Simply training on partial information isn't enough if the objective doesn't strongly regulate behavior. The entropy term in their PPO agent's loss function was insufficient to ensure robust behavior with very low information.

Key Results

  • The trained agents (PI-50) could achieve a win rate of 99% against themselves when provided with 50% of the color information.
  • In a round-robin tournament of 1000 games, DeepShap, GuidedBackprop, and the Frank-Wolfe method emerged as top performers, outperforming methods like Gradient, LRP, and SmoothGrad.
  • Shapley Sampling, while theoretically sound, showed weaker performance in identifying ground truth pieces compared to heuristic methods like DeepShap and GuidedBackprop.
  • The Frank-Wolfe method, while faster than Shapley Sampling, exhibited polarizing results, either identifying all or none of the important pieces.

Limitations and Future Work

  • Training objective for low information: The authors note that for very low information input, the policy network's behavior can become ill-defined, and the entropy loss might not be sufficient to regulate it towards a uniform distribution. They suggest exploring Q-learning or incorporating a "don't know" option as potential remedies.
  • Generalizability: While Connect Four is a good proxy, the authors acknowledge that the findings might not directly translate to more complex domains like image recognition.
  • Robustness to manipulation: Future work could focus on making saliency attribution methods more robust to manipulation, potentially by adapting techniques for handling off-manifold data.
  • Time efficiency of theoretically grounded methods: The paper highlights the computational cost of Shapley values and suggests exploring more efficient approximation methods.

Summary updated

Abstract

One of the goals of Explainable AI (XAI) is to determine which input components were relevant for a classifier decision. This is commonly know as saliency attribution. Characteristic functions (from cooperative game theory) are able to evaluate partial inputs and form the basis for theoretically "fair" attribution methods like Shapley values. Given only a standard classifier function, it is unclear how partial input should be realised. Instead, most XAI-methods for black-box classifiers like neural networks consider counterfactual inputs that generally lie off-manifold. This makes them hard to evaluate and easy to manipulate. We propose a setup to directly train characteristic functions in the form of neural networks to play simple two-player games. We apply this to the game of Connect Four by randomly hiding colour information from our agents during training. This has three advantages for comparing XAI-methods: It alleviates the ambiguity about how to realise partial input, makes off-manifold evaluation unnecessary and allows us to compare the methods by letting them play against each other.
BibTeX
@inproceedings{waldchen2022training,
  author = {Stephan Wäldchen and Felix Huber and Sebastian Pokutta},
  title = {{Training Characteristic Functions with Reinforcement Learning: XAI-methods play Connect Four}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=sPKogzLjD2B},
}
Figure