Explicable Policy Search

Ze Gong, Yu ZhangNeurIPS 2022 · OpenReview

Explicable Policy Search (EPS) is a reinforcement learning framework that allows agents to reconcile their optimal task performance with human expectations. By learning a "surrogate reward function" from human feedback, an agent can navigate stochastic environments in ways that make sense to a human observer—even when that human has a flawed or incomplete understanding of the environment’s physics.

The Model Reconciliation Problem

Imagine a drone navigating a gusty canyon. The drone knows there’s a heavy crosswind, so it flies in a zigzag pattern to stay stable. A human observer on the ground, unaware of the wind, sees this zigzagging as erratic and untrustworthy; they expect a straight line.

Motivating scenario and problem setting of EPS

This illustrates the core of the Model Reconciliation Problem. In human-agent teaming, a human’s mental model of the world and their reward preferences often differ from the agent’s true environment and engineered goals. Traditional Reinforcement Learning (RL) focuses solely on maximizing the agent's reward under its true environment, often resulting in "inexplicable" behaviors that degrade human trust. While "explicable planning" existed for simple, deterministic tasks, extending this to complex, stochastic, and continuous environments remained a significant challenge.

Explicable Policy Search (EPS)

The authors introduce Explicable Policy Search (EPS), which treats explicability as a multi-objective optimization problem.

  1. The Explicability Score: Instead of just measuring the physical distance between paths, EPS defines explicability as the negative KL-divergence between the distribution of trajectories the agent actually takes and the distribution the human expects it to take.
  2. The Surrogate Reward: Directly learning a human’s internal beliefs about physics and rewards is incredibly difficult. The authors prove that all the necessary information can be bundled into a single "surrogate reward function." This function is learned using Preference-based RL, where humans are shown two video clips of the agent and asked which behavior looks more expected.
  3. Modified Objective: The final learning objective combines the original task reward, the learned surrogate reward, and a novel Environment Entropy term. This entropy term encourages the agent to prefer parts of the environment where its actions are more likely to match human expectations by "default" due to the environment's inherent randomness.

Evaluation Domains

The researchers evaluated EPS using two primary setups:

Synthetic navigation domains used for evaluation
  • Synthetic Navigation Domains (D1-D4): Four custom-designed continuous 2D environments featuring obstacles like pits (high penalty), icy roads (slippery), and sandy roads (slow/costly). These tested the agent against "synthetic" human models with specific misunderstandings (e.g., the human thinks ice is dangerous when it’s actually fine for the agent).
  • Autonomous Driving Domain: A simulated highway environment where an ego-vehicle must handle a car ahead slowing down abruptly.
  • Human Subject Study: Data was collected from 15 participants to train the surrogate reward and validate the final policies.

Key Insights and Results

  • Reconciliation over Imitation: EPS doesn't blindly follow human expectations. If following an expectation would lead to a catastrophic failure (like driving into a pit), the agent prioritizes the task reward. It only "takes the scenic route" to be explicable when the cost to performance is acceptable. In "Domain 3," where human expectations directly conflicted with safety, EPS successfully ignored the dangerous expectation to ensure task success, whereas baseline methods failed.
  • Superior Explicability: In all synthetic domains, EPS achieved the highest explicability scores compared to standard RL (Soft Actor-Critic) and simple preference-based RL.
  • Human Preference: In the autonomous driving study, humans rated the EPS agent significantly higher (7.6/10) than the standard RL agent (5.0/10). The EPS agent chose to brake (meeting expectations) and then steer to safety, whereas the standard RL agent steered immediately—an efficient move that humans found "surprising" and unsafe.

Limitations and Future Work

  • Static Beliefs: The model assumes the human’s mental model is fixed. In reality, humans learn. Future work could involve "active" explicability where the agent helps the human update their mental model.
  • Computational Complexity: Learning the surrogate reward requires active human feedback, which can be expensive or slow to collect in complex real-time scenarios.
  • Sole Observer Assumption: The current framework assumes the human is just watching. Extending this to collaborative settings—where the human and agent are working together on the same task—is the next logical step.

Summary updated

Abstract

Human teammates often form conscious and subconscious expectations of each other during interaction. Teaming success is contingent on whether such expectations can be met. Similarly, for an intelligent agent to operate beside a human, it must consider the human’s expectation of its behavior. Disregarding such expectations can lead to the loss of trust and degraded team performance. A key challenge here is that the human’s expectation may not align with the agent’s optimal behavior, e.g., due to the human’s partial or inaccurate understanding of the task domain. Prior work on explicable planning described the ability of agents to respect their human teammate’s expectations by trading off task performance for more expected or “explicable” behaviors. In this paper, we introduce Explicable Policy Search (EPS) to significantly extend such an ability to stochastic domains in a reinforcement learning (RL) setting with continuous state and action spaces. Furthermore, in contrast to the traditional RL methods, EPS must at the same time infer the human’s hidden expectations. Such inferences require information about the human’s belief about the domain dynamics and her reward model but directly querying them is impractical. We demonstrate that such information can be necessarily and sufficiently encoded by a surrogate reward function for EPS, which can be learned based on the human’s feedback on the agent’s behavior. The surrogate reward function is then used to reshape the agent’s reward function, which is shown to be equivalent to searching for an explicable policy. We evaluate EPS in a set of navigation domains with synthetic human models and in an autonomous driving domain with a user study. The results suggest that our method can generate explicable behaviors that reconcile task performance with human expectations intelligently and has real-world relevance in human-agent teaming domains.
BibTeX
@inproceedings{gong2022explicable,
  author = {Ze Gong and Yu Zhang},
  title = {{Explicable Policy Search}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=82N_rasrUT_},
}
Figure