Explicable Policy Search (EPS) is a reinforcement learning framework that allows agents to reconcile their optimal task performance with human expectations. By learning a "surrogate reward function" from human feedback, an agent can navigate stochastic environments in ways that make sense to a human observer—even when that human has a flawed or incomplete understanding of the environment’s physics.
The Model Reconciliation Problem
Imagine a drone navigating a gusty canyon. The drone knows there’s a heavy crosswind, so it flies in a zigzag pattern to stay stable. A human observer on the ground, unaware of the wind, sees this zigzagging as erratic and untrustworthy; they expect a straight line.
This illustrates the core of the Model Reconciliation Problem. In human-agent teaming, a human’s mental model of the world and their reward preferences often differ from the agent’s true environment and engineered goals. Traditional Reinforcement Learning (RL) focuses solely on maximizing the agent's reward under its true environment, often resulting in "inexplicable" behaviors that degrade human trust. While "explicable planning" existed for simple, deterministic tasks, extending this to complex, stochastic, and continuous environments remained a significant challenge.
Explicable Policy Search (EPS)
The authors introduce Explicable Policy Search (EPS), which treats explicability as a multi-objective optimization problem.
- The Explicability Score: Instead of just measuring the physical distance between paths, EPS defines explicability as the negative KL-divergence between the distribution of trajectories the agent actually takes and the distribution the human expects it to take.
- The Surrogate Reward: Directly learning a human’s internal beliefs about physics and rewards is incredibly difficult. The authors prove that all the necessary information can be bundled into a single "surrogate reward function." This function is learned using Preference-based RL, where humans are shown two video clips of the agent and asked which behavior looks more expected.
- Modified Objective: The final learning objective combines the original task reward, the learned surrogate reward, and a novel Environment Entropy term. This entropy term encourages the agent to prefer parts of the environment where its actions are more likely to match human expectations by "default" due to the environment's inherent randomness.
Evaluation Domains
The researchers evaluated EPS using two primary setups:
- Synthetic Navigation Domains (D1-D4): Four custom-designed continuous 2D environments featuring obstacles like pits (high penalty), icy roads (slippery), and sandy roads (slow/costly). These tested the agent against "synthetic" human models with specific misunderstandings (e.g., the human thinks ice is dangerous when it’s actually fine for the agent).
- Autonomous Driving Domain: A simulated highway environment where an ego-vehicle must handle a car ahead slowing down abruptly.
- Human Subject Study: Data was collected from 15 participants to train the surrogate reward and validate the final policies.
Key Insights and Results
- Reconciliation over Imitation: EPS doesn't blindly follow human expectations. If following an expectation would lead to a catastrophic failure (like driving into a pit), the agent prioritizes the task reward. It only "takes the scenic route" to be explicable when the cost to performance is acceptable. In "Domain 3," where human expectations directly conflicted with safety, EPS successfully ignored the dangerous expectation to ensure task success, whereas baseline methods failed.
- Superior Explicability: In all synthetic domains, EPS achieved the highest explicability scores compared to standard RL (Soft Actor-Critic) and simple preference-based RL.
- Human Preference: In the autonomous driving study, humans rated the EPS agent significantly higher (7.6/10) than the standard RL agent (5.0/10). The EPS agent chose to brake (meeting expectations) and then steer to safety, whereas the standard RL agent steered immediately—an efficient move that humans found "surprising" and unsafe.
Limitations and Future Work
- Static Beliefs: The model assumes the human’s mental model is fixed. In reality, humans learn. Future work could involve "active" explicability where the agent helps the human update their mental model.
- Computational Complexity: Learning the surrogate reward requires active human feedback, which can be expensive or slow to collect in complex real-time scenarios.
- Sole Observer Assumption: The current framework assumes the human is just watching. Extending this to collaborative settings—where the human and agent are working together on the same task—is the next logical step.
Summary updated