Introduction
As autonomous agents—from self-driving cars to medical assistants—become increasingly integrated into society, the ability for humans to accurately compare and select between competing AI policies is crucial. Traditional explanation methods, such as the HIGHLIGHTS algorithm, generate summaries for agents independently by showcasing behavior in "important" states. However, this approach often fails when comparing high-performing agents that share similar "highlights" but diverge significantly in critical edge cases. This paper introduces DISAGREEMENTS, an algorithm designed to generate contrastive summaries that highlight exactly where two reinforcement learning (RL) policies differ, thereby improving human decision-making.
The DISAGREEMENTS Algorithm
The core innovation of the DISAGREEMENTS algorithm is the shift from independent to contrastive summarization. Instead of looking at one agent in isolation, the algorithm identifies states where two agents—a "Leader" and a "Disagreer"—would take different actions.
The methodology involves five key steps:
- Parallel Execution: Simulating both agents simultaneously in the same environment.
- Divergence Identification: Marking states where the agents' preferred actions differ ($π_1(s) \neq π_2(s)$).
- Branching Trajectories: At a point of disagreement, the Disagreer "splits off" to show the outcome of its alternative action relative to the Leader's path.
- Importance Ranking: Using Last State Importance (normalized Q-value differences) to prioritize disagreements that lead to significantly different outcomes rather than trivial variations.
- Visualization: Presenting side-by-side video clips of the diverging paths to the user.
Experimental Evaluation
The researchers evaluated DISAGREEMENTS against the state-of-the-art HIGHLIGHTS method using two distinct environments:
- Frogger: Used to test superiority identification between agents of varying skill levels: Expert (E), Mid-range (M), and LimitedVision (LV).
- Highway: Used to test the ability to convey behavioral traits (e.g., speed vs. safety) among agents with different reward functions like "ClearLane" and "SocialDistance."
Key Results
User studies conducted via Amazon Mechanical Turk demonstrated that contrastive summaries significantly outperform independent ones in helping humans distinguish agent capabilities and personalities.
In the Frogger "Expert vs. Limited Vision" matchup, the success rate for identifying the better agent rose from roughly 30% with HIGHLIGHTS to nearly 80% with DISAGREEMENTS ($p < 1.6 \times 10^{-5}$). Similar improvements were observed in the Highway environment, where users were significantly better able to identify specific behavioral priorities, such as lane preference ($p = 0.014$).
A critical insight from the study was that independent summaries can inadvertently hide weaknesses. Participants viewing HIGHLIGHTS often perceived weaker agents as "more decisive" because they only saw successful moments, whereas DISAGREEMENTS explicitly exposed the failures of the weaker policy in states where the superior agent succeeded.
Discussion and Limitations
While highly effective, the DISAGREEMENTS algorithm currently focuses on pairwise comparisons. Scaling this to multiple agents without causing cognitive overload remains a challenge. Additionally, the method requires access to an interactive simulator and the agents' internal Q-values, which may not always be available in "black-box" scenarios. Future work will explore categorizing "disagreement types" (e.g., safety vs. speed) to provide even more nuanced explanations of behavioral divergence.
Summary updated