Exploring Computational User Models for Agent Policy Summarization

Isaac Lage, Daphna Lifschitz, Finale Doshi-Velez, Ofra Amir

Introduction

As AI agents are increasingly deployed in high-stakes domains like autonomous driving and medical treatment, the need for human-interpretable explanations of their behavior becomes paramount. Policy summarization aims to provide this understanding by selecting a representative set of state-action pairs that convey the agent's global logic. Traditionally, researchers have assumed that humans act as Inverse Reinforcement Learning (IRL) engines—inferring underlying rewards to predict future behavior. However, cognitive science suggests that humans may instead rely on Imitation Learning (IL), using simple pattern matching based on situational similarity. This paper investigates how the choice of computational user model during summary extraction affects a human's ability to reconstruct and predict an agent's policy.

Methodology

The study evaluates two primary frameworks for both summary extraction (selecting examples) and policy reconstruction (predicting behavior from those examples):

  1. IRL-Based Models: These assume the user seeks to recover the agent's reward function. The authors utilize the Set Cover Optimal Teaching (SCOT) algorithm to identify the minimal set of demonstrations that uniquely define an optimal policy.
  2. IL-Based Models: These assume the user maps states directly to actions based on similarity. A novel extraction method was developed using Gaussian Random Fields (GRF) combined with an Active Learning approach to select the most informative states.

A "cross-over" experimental design was employed to measure reconstruction accuracy when the extraction model either matches or mismatches the reconstruction model.

Experimental Setup

The frameworks were tested across three distinct simulated environments:

  • Random Gridworld: A navigation task involving movement toward colored tiles with varying rewards.
  • PAC-MAN: A dynamic environment requiring the agent to balance food collection with ghost avoidance.
  • HIV Simulator: A complex medical domain where agents decide on drug administration based on six physiological biomarkers.

Results

The computational simulations revealed a clear "Matching Principle": reconstruction accuracy is highest when the extraction and reconstruction models are aligned.

Accuracy and Value Difference heatmaps across three domains. The diagonal patterns highlight that matching the extraction and reconstruction models leads to superior policy recovery.

Key findings from the human studies include:

  • Domain-Specific Mental Models: Human reasoning strategies vary by task. In the unfamiliar HIV domain, 78% of participants favored IL-style pattern matching. Conversely, in the more intuitive Gridworld, users leaned toward IRL-style goal-seeking.
  • Performance Gains from Alignment: In the HIV domain, participants provided with IL-optimized summaries achieved 45% prediction accuracy, significantly outperforming those given IRL summaries (33%).
  • IL Sufficiency: In dynamic or signal-heavy environments like PAC-MAN and HIV, IL-based summaries often yielded higher reconstruction accuracy than IRL-based ones, suggesting that teaching rewards is not always the optimal explanatory strategy.

Discussion and Conclusion

The research highlights that failing to account for a user's actual mental model can significantly hinder communication. While IRL is often the default assumption in explainable AI, IL is frequently more effective, especially in complex or unfamiliar domains.

Future work should address the sensitivity of these models to feature representation, as human perceptions of "similarity" may differ from mathematical kernels. Additionally, exploring hybrid summaries or adaptive systems that learn a user's mental model in real-time could further enhance the effectiveness of agent policy summarization.

Summary updated

Abstract

AI agents are being developed to support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.
BibTeX
@inproceedings{lage2019exploring,
  author = {Isaac Lage and Daphna Lifschitz and Finale Doshi-Velez and Ofra Amir},
  title = {{Exploring Computational User Models for Agent Policy Summarization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/194},
}
Figure