Introduction
Deep Reinforcement Learning (DRL) models often operate as "black boxes," making it difficult for users to understand the reasoning behind an agent's decisions. While existing research focuses on explaining single-agent behavior, a significant gap exists in comparing multiple competent agents. This is particularly relevant in domains like autonomous driving, where a user might need to choose between different driving styles (e.g., aggressive vs. cautious).
This paper introduces a novel framework to explain "stylistic" differences between two RL agents. The core challenge is distinguishing between differences in ability (one agent is simply better or more trained) and preference (both agents are competent but prioritize different trade-offs).
Methodology
The proposed framework generates contrastive natural-language explanations through a three-step process:
- Disagreement Data Collection: Two policies, $\pi_A$ and $\pi_B$, are "unrolled" in the same environment. When they choose different actions in the same state, the system records the state and simulates both agents for $k$ future steps to observe the resulting outcomes.
- Filtering for Preferences: To isolate stylistic choices from errors, the system applies three filters: both agents must be highly confident in their actions, they must perceive the starting state as having similar potential value ($V(s)$), and they must be similarly satisfied with their respective outcomes after $k$ steps.
- Explanation Generation: The system performs a paired T-test on state features (e.g., speed, distance) across the outcomes. Significant differences are then mapped to a natural language template (e.g., "Policy A prefers states with smaller velocity...").
Experimental Setup: Autonomous Merging
The researchers evaluated their approach using a simulated autonomous driving merging task. In this scenario, an agent must merge into a lane occupied by a non-autonomous vehicle.
The study utilized a baseline "safety-oriented" agent and several "speed-oriented" agents trained with varying reward weights for progress. An under-trained "inferior" agent was also included to test the system's ability to filter out ability-based differences.
Key Results
The framework demonstrated high specificity and sensitivity in identifying preference-based differences. By requiring high confidence and similar state valuations, the authors isolate "intentional" stylistic choices from noise or incompetence.
- Specificity: When comparing agents with identical preferences or comparing a skilled agent to a random one, the method correctly identified zero preference-based disagreements. This proves the filtering mechanism effectively ignores "clueless" behavior.
- Sensitivity: When comparing safety-oriented vs. speed-oriented agents, the system identified hundreds of valid preference-based disagreements.
- Interpretability: The system produced accurate summaries that matched the ground-truth reward functions used to train the agents, such as identifying preferences for smaller velocities in safety-oriented policies.
Conclusion and Future Work
By focusing on "intentional" stylistic choices rather than simple action differences, this method provides a more nuanced understanding of RL agent behavior. Future work will address the manual tuning of filtering hyperparameters and extend the framework to compare larger fleets of agents. Additionally, improving the interpretability of abstract input features remains a key challenge for broader application.
Summary updated