Explanation-Guided Reward Alignment

Saaduddin Mahmud, Sandhya Saisubramanian, Shlomo ZilbersteinIJCAI 2023 · DOI

TL;DR

This paper introduces REVEALE, a framework that uses explanations to verify and improve how well an AI agent's learned reward function aligns with human intent. By having an agent explain its reasoning and a human provide feedback on these explanations, REVEALE can detect and correct misaligned rewards, leading to safer and more reliable AI behavior, especially in novel situations.

What Problem Does This Paper Address?

Autonomous agents are increasingly tasked with complex real-world jobs, often learning their objectives through demonstrations. Inverse Reinforcement Learning (IRL) is a common technique to infer a reward function from these demonstrations. However, a significant challenge arises because the inferred reward function might not perfectly match the original human intent. This misalignment can happen due to incomplete or biased demonstration data, leading to spurious correlations between features and rewards. For instance, an autonomous vehicle trained to stop for pedestrians with dogs might fail to stop for a pedestrian alone if dogs are always present in the training data. Operating with such misaligned rewards can lead to unsafe or undesirable behavior. Furthermore, black-box reward representations make it difficult to verify their correctness and prevent harmful actions.

Methods

REVEALE tackles this problem through a two-phase approach: verification and improvement.

  1. Reward Inference: The process begins by inferring an initial posterior distribution over possible reward functions from the demonstration data using a Bayesian IRL method (specifically, BREX).
  2. Verification Phase: The agent is prompted to explain its learned reward for specific "verification test states." These explanations consist of the reward value at that state and feature attributions (e.g., saliency maps, gradients) indicating which state features influenced the reward. A human tester then evaluates these explanations.
  3. If the explanation is approved, it reinforces the current reward.
  4. If it fails, the agent provides alternative explanations from other candidate reward functions. The tester then provides feedback, either by selecting the most accurate explanation or by ranking explanations.
  5. Improvement Phase: The feedback gathered from the tester is used to update the posterior distribution of reward functions. This refined posterior is then used to calculate a new, more aligned reward function, and the process iterates until the reward function passes all verification tests.

The key novelty lies in using explanations as the medium for human feedback, allowing for verification and correction of reward alignment even when the demonstration data is insufficient to cover all scenarios.

Datasets

The paper evaluates REVEALE across five proof-of-concept domains:

  • WaterWorld: A linear reward domain testing response to distribution shifts, where water locations change between training and testing.
  • DogWalk: A linear reward domain where an AV must learn to stop for pedestrians and dogs, with potential spurious correlations.
  • LavaLand: A linear reward domain with terrain types, where the "lava" indicator might be absent in training data, leading to potential unsafe behavior.
  • AVNav (Autonomous Vehicle Navigation): A non-linear reward domain for safe route planning, featuring spurious correlations between good mobile network quality and accident history.
  • CoinRush: A non-linear reward domain involving collecting coins and avoiding enemies, where enemy/coin colors are correlated in training but not in testing.

The training data for these domains is generated by sub-optimally solving training instances, and the learned rewards are evaluated on test instances that differ in start states and risky region locations.

Key Insights and Ideas

  • Explanations as a Bridge: The core insight is that explanations, not just reward values or policies, can serve as a powerful interface for humans to verify and correct AI behavior. This is particularly useful when direct demonstrations for all scenarios are infeasible.
  • Detecting and Correcting Misalignment: REVEALE effectively identifies reward functions that are consistent with the training data but would lead to undesirable behavior in novel situations. It then actively corrects this misalignment.
  • Beyond Data Quantity: The paper demonstrates that simply increasing the size of the demonstration dataset doesn't guarantee improved reward alignment. Alternative feedback mechanisms, like explanations, are crucial.
  • Feature Attribution Matters: The choice of explanation technique (e.g., gradient-based vs. saliency maps) impacts the effectiveness of feedback, with gradient-based methods often proving more informative for alignment.

Key Results

  • Improved Prediction Accuracy: REVEALE, especially with ranked GaE explanations, achieved the highest accuracy in predicting trajectory rankings on test environments, significantly outperforming traditional IRL methods like REX.
  • Better Reward Estimation: In novel states, REVEALE led to more proportional reward estimation compared to REX, which often over- or underestimated rewards.
  • Safer Policies: In domains with risky states (LavaLand, DogWalk, WaterWorld), REVEALE-trained agents demonstrated a significantly higher percentage of avoiding risky regions, leading to safer execution.
  • Efficiency of Feedback: The paper shows that a relatively small amount of explanation feedback (e.g., 64 explanations for linear rewards) can lead to maximal accuracy.

Limitations and Future Work

  • Limitations:
    • The effectiveness of REVEALE can depend on the quality of the explanation generation method and the tester's ability to provide accurate feedback.
  • While REVEALE can reduce reward ambiguity, achieving complete optimality might not always be feasible, especially with less informative feedback types like saliency maps.
  • The current framework assumes the intended reward function can be represented within the model's capacity (e.g., linear reward functions are assumed to be representable by a vector in Rⁿ).
  • Future Work:
    • Develop techniques to automatically identify critical states for verification.
  • Integrate REVEALE with active learning methods to minimize the number of required feedback queries.
  • Explore the efficiency of other explanation methods, particularly causal explanations, for reward alignment.

Summary updated

BibTeX
@inproceedings{mahmud2023explanation,
  author = {Saaduddin Mahmud and Sandhya Saisubramanian and Shlomo Zilberstein},
  title = {{Explanation-Guided Reward Alignment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/53},
}
Figure