Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)

Tobias Huber, Katharina Weitz, Elisabeth André, Ofra AmirIJCAI 2022 · DOI

TL;DR

This paper explores how to make AI agents more understandable by combining "global" strategy summaries with "local" saliency maps that highlight what the agent is looking at. Their user study found that well-chosen strategy summaries significantly improve users' ability to assess agent performance, while the impact of saliency maps was more nuanced, suggesting a need for better accessibility and integration.

What Problem Does This Paper Address?

As Artificial Intelligence (AI) agents become more sophisticated and are deployed in high-stakes domains like healthcare and transportation, understanding their decision-making processes is paramount. Deep Reinforcement Learning (RL) agents, in particular, operate in complex environments with vast state spaces and delayed rewards, making their behavior difficult to interpret. This paper tackles the challenge of explaining these agents' actions by investigating how to combine two types of explanations: global explanations that describe the agent's overall strategy across various situations, and local explanations that pinpoint what information the agent focuses on in a specific moment. The goal is to create a more comprehensive and intuitive understanding for human users.

Methods

The researchers propose a novel approach that integrates two explanation techniques:

  • Strategy Summaries (Global Explanation): They adapt the HIGHLIGHTS-DIV algorithm, which generates summaries by selecting "important" states that showcase an agent's behavior across a range of scenarios. The key innovation here is the intelligent selection of these states to represent the agent's overall strategy, aiming for diversity and informativeness.
  • Saliency Maps (Local Explanation): These maps, generated using Layer-Wise Relevance Propagation (LRP), highlight the specific input features (e.g., pixels in a game) that the agent attends to when making a decision in a particular state.

The core contribution is the combination of these two methods, where saliency maps are overlaid onto the states presented in the strategy summaries.

Datasets

The experiments were conducted using the Atari 2600 game Ms. Pac-Man, implemented within the Arcade Learning Environment (ALE). The agents were trained using the OpenAI baselines implementation of the DQN algorithm. For evaluation, streams of 10,000 game steps were recorded for each trained agent. No new benchmarks were introduced; the focus was on evaluating explanation methods within a well-established domain.

Key Insights and Ideas

  • Global Strategy Matters Most: The study strongly suggests that the selection of states for global strategy summaries is more critical for user understanding than the inclusion of local saliency maps.
  • HIGHLIGHTS-DIV Outperforms Likelihood-Based Summaries: Summaries generated by the HIGHLIGHTS-DIV algorithm, which intelligently selects important states, led to better user performance in assessing agent strategies and comparing agent capabilities compared to summaries based on simply sampling likely states.
  • Saliency Maps Have Nuanced Impact: While saliency maps did not significantly improve performance on simple object identification tasks, they showed a positive trend in helping users describe agent strategies textually, suggesting they can help contextualize why certain objects are important.
  • Accessibility is Key for Saliency Maps: Participants struggled with interpreting saliency maps, finding them non-intuitive and sometimes difficult to track, especially when presented dynamically within a video. This suggests that usability barriers need to be addressed for saliency maps to be more effective.
  • Combination Potential: The paper posits that the benefits of saliency maps are likely amplified when combined with well-chosen global explanations, suggesting a promising avenue for future research in integrated explanation frameworks.

Key Results

A user study with 133 participants was conducted across four experimental conditions (Likelihood-based Summaries (L), HIGHLIGHTS-DIV Summaries (H), L+Saliency (L+S), and H+Saliency (H+S)).

  • Retrospection Task (Understanding Agent Strategy): Participants in the HIGHLIGHTS-DIV conditions (H and H+S) achieved significantly higher scores and reported greater satisfaction compared to the likelihood-based conditions (L and L+S). Condition H+S showed a trend towards being the most helpful for textual strategy descriptions.
  • Agent Comparison Task (Assessing Performance/Trust): Participants in condition H performed significantly better than in condition L. There was also a positive trend for L+S compared to L.
  • Saliency Map Impact: No significant differences were found between saliency and non-saliency conditions in overall performance, but saliency maps did contribute to improved textual strategy descriptions in the H+S condition.

Limitations and Future Work

  • Saliency Map Usability: The authors acknowledge significant usability barriers with saliency maps, including their non-intuitive nature for non-experts and the difficulty in tracking attention within dynamic video summaries.
  • Accessibility Improvements Needed: Future work should focus on addressing these accessibility issues to unlock the full potential of saliency maps.
  • Combined Framework Potential: The paper strongly advocates for the development of combined explanation frameworks, where the strengths of global strategy summarization and local attention highlighting can be synergistically leveraged. This is particularly relevant as many existing local explanation methods implicitly rely on a global decision of which states to present.
  • Implicit vs. Explicit Trust: While the study measured appropriate trust indirectly through agent selection, future work could explore more direct measures of trust.

Summary updated

BibTeX
@inproceedings{huber2022local,
  author = {Tobias Huber and Katharina Weitz and Elisabeth André and Ofra Amir},
  title = {{Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/803},
}
Figure