TL;DR
In human-AI teams, knowing exactly when to override an autonomous agent is critical: intervene too little, and the agent might fail dangerously; intervene too much, and the system loses its utility. This paper introduces the Termination Markov Decision Process (TerMDP) framework and develops an algorithmic policy-summarization method that selects a handful of key trajectories to help human supervisors accurately predict agent failures and optimize interventions.
What Problem Does This Paper Address?
As AI agents are increasingly deployed in sequential decision-making tasks—like autonomous driving, factory automation, and healthcare—they often operate under the watchful eye of a human supervisor. In these setups, the human acts as a "terminator" who can temporarily take over control when the agent is about to make a mistake.
To intervene optimally, the human needs an accurate mental model of the agent's capabilities and blind spots. However, deep reinforcement learning (RL) policies operate in massive state spaces and are driven by complex, long-term reward optimizations, making them notoriously opaque.
Prior work in Explainable RL (XRL) often relies on local explanations (like saliency maps showing what the AI is looking at in a single frame). But in fast-moving, real-time environments, a human cannot analyze explanations frame-by-frame. Instead, they need a global explanation—a high-level summary of the policy before the task begins. This paper addresses the challenge of automatically generating policy summaries specifically tailored to help humans decide when to take over.
Methods
The authors formalize the problem using a new framework called the Termination Markov Decision Process (TerMDP), which extends standard MDPs by incorporating a human policy ($\pi^{\text{human}}$), a take-over horizon ($h$ steps), and an intervention cost ($c$).
The workflow is split into two distinct phases:
To select the optimal subset of $K$ trajectories for the summary, the authors propose three distinct algorithmic scoring schemes:
- Score 1: Reducing Uncertainty: This method trains an ensemble of Deep Q-Networks (DQNs) on the agent's history. It identifies states where the ensemble shows high variance in its action-value predictions (indicating complex or rare scenarios) and selects trajectories that minimize this variance when added to the summary's training set.
- Score 2: Minimizing Q-Value Reconstruction Error: This scheme selects trajectories that minimize the squared difference between the expected Q-values of the true agent policy and a proxy policy trained purely on the summary data.
- Score 3: Maximizing Policy Likelihood: Instead of estimating value functions, this approach treats the summary as an imitation learning dataset. It trains an XGBoost classifier to predict the agent's actions from states using only the summary trajectories, optimizing for the average log-likelihood of the model on a held-out evaluation set.
Optimization: Finding the absolute best $K$ trajectories out of a massive dataset is a combinatorially hard problem. To circumvent this, the authors implement a Genetic Algorithm that treats subsets of trajectories as chromosomes, evolving them over generations to maximize the chosen scoring metric.
Datasets
The methodology was implemented and evaluated using Highway-Env, a widely used autonomous driving simulator.
- Agent Training: A standard DQN agent was trained to navigate highway traffic.
- Flawed Policy Injection: To create a realistic scenario where human intervention is actually required, the authors engineered a hybrid agent. It behaves optimally across three out of four lanes, but if it enters the leftmost lane, it switches to a "degenerate" policy that only executes a single action (either continuous acceleration or deceleration), inevitably requiring human takeover to avoid failure.
Key Insights and Ideas
- Reconstruction vs. Utility: Standard policy summarization methods often try to minimize reconstruction error uniformly across the entire state space. The authors argue this is sub-optimal for supervision. For example, showing a self-driving car driving perfectly on an empty road is a waste of a summary slot; summaries must focus on high-uncertainty or high-risk states where human intervention changes the outcome.
- The Human "Collision Bias": During an initial pilot proxy study, the authors discovered a fascinating cognitive bias: if a summary trajectory contained a collision, human participants heavily anchored on it. They assumed the agent was generally reckless, completely ignoring speed profiles or lane-changing nuances. This suggests that purely mathematical optimization metrics must eventually be paired with cognitive heuristics to counteract human biases.
Key Results
- Algorithmic Convergence: The genetic algorithm demonstrated consistent convergence across all three scoring metrics and varying summary lengths ($K = 3, 5, 7, 9$ episodes).
- Improved Human Intervention: In a user study ($n = 160$) evaluating Score 3 (Policy Likelihood) with a 5-episode summary, participants who viewed the algorithmically optimized summaries significantly outperformed those who viewed low-scoring summaries ($p\text{-value} = 0.02$). They successfully identified the agent's flawed leftmost-lane behavior and made significantly better choices on when to terminate the agent's control.
Limitations and Future Work
- Untested Metrics: While three scoring metrics were mathematically formulated, the human user study exclusively evaluated Score 3 (Policy Likelihood). The real-world human efficacy of the uncertainty-reduction and Q-reconstruction metrics remains unproven.
- Domain Simplicity: The evaluation was restricted to a simulated driving environment with a highly structured, single-lane policy flaw. Future work is required to see how well these global summaries scale to more complex, multi-modal agent flaws and diverse domains like medical robotics or industrial automation.
- Accounting for Cognitive Biases: Given the discovery of the human "collision bias," the authors note that future iterations of the summarizer module will need to explicitly incorporate human psychological constraints rather than relying solely on machine-learning performance metrics.
Summary updated