TL;DR
This paper introduces COViz, a novel method for explaining Reinforcement Learning (RL) agent decisions by visualizing the outcomes of alternative actions. They demonstrate that combining COViz with reward decomposition significantly improves users' understanding of agent preferences, even though users subjectively preferred COViz alone.
What Problem Does This Paper Address?
Reinforcement Learning (RL) agents learn complex behaviors through trial and error, often making decisions in ways that are opaque to humans. This "black box" nature makes it difficult to trust these agents, especially in high-stakes applications like healthcare or autonomous driving. Existing methods for explaining RL agents (Explainable RL or XRL) often focus on why an agent made a specific decision in a given state (local explanations), but they typically don't show what would have happened if the agent had chosen differently. This gap makes it hard for users to truly grasp the agent's preferences and the trade-offs it considered.
Methods
The core contribution is COViz (Counterfactual Outcome Visualization). COViz addresses the "what if" question by:
- Generating Counterfactual Trajectories: For a given state and the agent's chosen action, COViz simulates the outcome of an alternative, "counterfactual" action for a short duration (k steps). This alternative action is typically the agent's second-best choice.
- Visual Comparison: The true trajectory (from the chosen action) and the counterfactual trajectory are presented side-by-side, allowing users to visually compare the consequences of different choices. This contrastive approach is inspired by cognitive science, which suggests people understand explanations better when they contrast alternatives.
The authors also propose a Global Extension to COViz that integrates these local explanations into policy summaries by selecting important states based on counterfactual outcomes.
Furthermore, they investigate Integrating COViz with Reward Decomposition. Reward decomposition explains an agent's decision by breaking down its expected reward into different components (e.g., reward for speed vs. reward for not crashing). By combining COViz (showing outcomes) with reward decomposition (showing expected utility), they aim to provide a more comprehensive explanation.
Datasets
The paper uses a highway environment for training and evaluation. This environment allows an RL agent to control a green vehicle navigating a multi-lane highway, interacting with other vehicles. The environment is designed to be relatable and allows for agents with divergent, yet reasonable, driving strategies without a single "optimal" behavior, making it suitable for studying explanations of preferences. No new datasets were introduced; the focus was on evaluating explanation methods within this existing benchmark.
Key Insights and Ideas
- Counterfactuals are Key to Understanding Preferences: Simply knowing why an agent chose an action isn't enough. Seeing the outcomes of alternative actions is crucial for understanding the agent's underlying preferences and the trade-offs it makes.
- Integration is More Than the Sum of its Parts: Combining COViz (outcomes) with reward decomposition (motivations) significantly boosts users' ability to correctly identify agent preferences, even though it might increase cognitive load.
- Subjective Preference vs. Objective Performance: Users might prefer a simpler explanation (like COViz alone) but perform better when given a more complex, integrated explanation (COViz + Reward Decomposition). This highlights a common challenge in XAI evaluation.
- Domain Sensitivity: The effectiveness of certain state selection methods for generating counterfactuals can be domain-dependent.
Key Results
- Improved Performance with Integration: In both user studies, the integrated approach (CORD - COViz + Reward Decomposition) led to significantly higher correctness rates in participants' ability to assess agent preferences compared to COViz or Reward Decomposition alone.
- Study 1: CORD achieved a correctness rate of 0.64, compared to 0.42 for COViz and 0.45 for RD.
- Study 2: CORD achieved 0.63, compared to 0.40 for COViz and 0.48 for RD.
- COViz Preferred Subjectively: Despite the performance gains from integration, participants in Study 2 ranked COViz as their preferred explanation type (57% ranked it first), while the integrated method was ranked last by 53% of participants.
- Confidence and Satisfaction: Participants reported high confidence and satisfaction ratings across all conditions, with no significant differences between them.
Limitations and Future Work
- Single Counterfactual Action: The current COViz method enforces only a single counterfactual action deviation. In complex environments, a single deviation might be easily reversible or negligible. Future work could explore enforcing a sequence of counterfactual actions.
- Heuristic Counterfactual Selection: The choice of the counterfactual action (e.g., second-best) was heuristic. Future work could allow users to select counterfactual actions to increase interactivity and trust.
- User Cognitive Load: The integrated method, while improving performance, might overwhelm users. Future work could explore ways to manage cognitive load, such as cascading information or allowing user control over detail levels.
- Broader Task and Explanation Type Exploration: This work focused on identifying agent preferences. Future research should explore which explanation types are best suited for different user tasks and investigate combinations with other XRL methods like saliency maps or action graphs.
Summary updated