Abstract: AI agents are being developed to support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.
Introduction
As AI agents enter high-stakes fields like autonomous driving and medicine, humans need to understand their "policy"—the logic behind their decisions. Since it is impossible to inspect every state an agent might encounter, researchers use policy summarization: selecting a handful of representative state-action pairs to demonstrate behavior to a user.
Historically, the prevailing assumption was that humans act as "rational planners" using Inverse Reinforcement Learning (IRL), inferring an agent's goals (rewards) to extrapolate behavior. However, cognitive science suggests humans often rely on Imitation Learning (IL), matching new situations to the most similar examples previously observed. This paper investigates whether choosing the wrong "user model" when creating a summary renders that summary ineffective.
Methodology
The authors frame policy summarization as a machine teaching problem: identifying a set of $k$ examples that helps a specific "learner" model reconstruct the full policy most accurately. They compare two primary approaches:
- IRL-Based Extraction: Utilizes the SCOT algorithm and Maximum Entropy IRL, assuming the user seeks to learn the agent's underlying reward function.
- IL-Based Extraction: A novel method using Gaussian Random Fields (GRF). This model assumes the user predicts actions by propagating labels through a similarity graph. An active learning strategy (expected error reduction) is used to select the most informative states for this learner.
Success is measured via reconstruction quality: the accuracy with which a model (or human) predicts the agent's action in an unseen state.
Key Insights: The Mismatch Penalty
A central finding of the research is the "mismatch penalty." In computational simulations across Random Gridworld, PAC-MAN, and an HIV Simulator, the highest reconstruction accuracy consistently occurred when the extraction model matched the reconstruction model.
If a summary is optimized for an IL learner, an IRL learner performs poorly on it, and vice versa. This confirms that there is no "universally optimal" summary; effectiveness is relative to the user's mental model. Furthermore, the study found that context dictates strategy. In complex, non-navigational domains like the HIV Simulator, 78% of participants defaulted to Imitation Learning. In simpler navigational tasks like Gridworld, participants were more split, though those who successfully employed IRL achieved higher accuracy.
Experimental Results
Simulation Success
Across all three domains, the researchers observed that performance metrics improved as the trajectory length (the number of examples in the summary) increased, provided the models were aligned.
Human-Subject Study
- HIV Domain: Participants shown an IL-optimized summary achieved significantly higher accuracy (0.45) than those shown an IRL-optimized summary (0.33). This aligns with the finding that humans naturally use IL for unfamiliar, signal-based tasks.
- Gridworld Domain: While there was no significant difference between summary types overall, participants who reported using an IRL strategy were much more accurate (0.66) than those who used IL (0.27). This suggests that for navigational tasks, IRL is a more powerful—but more cognitively demanding—mental model.
Conclusion and Future Work
The research highlights that humans do not have a monolithic way of interpreting AI behavior. Complexity breeds a reliance on similarity-based reasoning (IL), while structured environments may allow for goal-based reasoning (IRL).
Future work should explore "human-in-the-loop" summarization. By observing a user's initial predictions, a system could infer their mental model in real-time and adapt the summary to match their specific reasoning style, ensuring more effective human-AI alignment.
Summary updated