Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agents

Jasmina Gajcin, Rahul Nair, Tejaswini Pedapati, Radu Marinescu, Elizabeth Daly, Ivana DusparicarXiv 2021 · arXiv

Introduction

Deep Reinforcement Learning (DRL) models often operate as "black boxes," making it difficult for users to understand the reasoning behind an agent's decisions. While existing research focuses on explaining single-agent behavior, a significant gap exists in comparing multiple competent agents. This is particularly relevant in domains like autonomous driving, where a user might need to choose between different driving styles (e.g., aggressive vs. cautious).

This paper introduces a novel framework to explain "stylistic" differences between two RL agents. The core challenge is distinguishing between differences in ability (one agent is simply better or more trained) and preference (both agents are competent but prioritize different trade-offs).

Methodology

The proposed framework generates contrastive natural-language explanations through a three-step process:

  1. Disagreement Data Collection: Two policies, $\pi_A$ and $\pi_B$, are "unrolled" in the same environment. When they choose different actions in the same state, the system records the state and simulates both agents for $k$ future steps to observe the resulting outcomes.
  2. Filtering for Preferences: To isolate stylistic choices from errors, the system applies three filters: both agents must be highly confident in their actions, they must perceive the starting state as having similar potential value ($V(s)$), and they must be similarly satisfied with their respective outcomes after $k$ steps.
  3. Explanation Generation: The system performs a paired T-test on state features (e.g., speed, distance) across the outcomes. Significant differences are then mapped to a natural language template (e.g., "Policy A prefers states with smaller velocity...").

Experimental Setup: Autonomous Merging

The researchers evaluated their approach using a simulated autonomous driving merging task. In this scenario, an agent must merge into a lane occupied by a non-autonomous vehicle.

The autonomous driving merging task illustrates two distinct stylistic preferences: merging behind or merging in front of a vehicle.

The study utilized a baseline "safety-oriented" agent and several "speed-oriented" agents trained with varying reward weights for progress. An under-trained "inferior" agent was also included to test the system's ability to filter out ability-based differences.

Key Results

The framework demonstrated high specificity and sensitivity in identifying preference-based differences. By requiring high confidence and similar state valuations, the authors isolate "intentional" stylistic choices from noise or incompetence.

Quantitative evaluation of the filtering mechanism, showing that preference-based disagreements are only identified when comparing agents with distinct, competent strategies.
  • Specificity: When comparing agents with identical preferences or comparing a skilled agent to a random one, the method correctly identified zero preference-based disagreements. This proves the filtering mechanism effectively ignores "clueless" behavior.
  • Sensitivity: When comparing safety-oriented vs. speed-oriented agents, the system identified hundreds of valid preference-based disagreements.
  • Interpretability: The system produced accurate summaries that matched the ground-truth reward functions used to train the agents, such as identifying preferences for smaller velocities in safety-oriented policies.

Conclusion and Future Work

By focusing on "intentional" stylistic choices rather than simple action differences, this method provides a more nuanced understanding of RL agent behavior. Future work will address the manual tuning of filtering hyperparameters and extend the framework to compare larger fleets of agents. Additionally, improving the interpretability of abstract input features remains a key challenge for broader application.

Summary updated

Abstract

In complex tasks where the reward function is not straightforward and consists of a set of objectives, multiple reinforcement learning (RL) policies that perform task adequately, but employ different strategies can be trained by adjusting the impact of individual objectives on reward function. Understanding the differences in strategies between policies is necessary to enable users to choose between offered policies, and can help developers understand different behaviors that emerge from various reward functions and training hyperparameters in RL systems. In this work we compare behavior of two policies trained on the same task, but with different preferences in objectives. We propose a method for distinguishing between differences in behavior that stem from different abilities from those that are a consequence of opposing preferences of two RL agents. Furthermore, we use only data on preference-based differences in order to generate contrasting explanations about agents' preferences. Finally, we test and evaluate our approach on an autonomous driving task and compare the behavior of a safety-oriented policy and one that prefers speed.
BibTeX
@misc{gajcin2021contrastive,
  author = {Jasmina Gajcin and Rahul Nair and Tejaswini Pedapati and Radu Marinescu and Elizabeth Daly and Ivana Dusparic},
  title = {{Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agents}},
  year = {2021},
  eprint = {2112.09462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.09462},
}
Figure