When an AI agent makes a decision that surprises a human user, the confusion usually stems from a mismatch in their internal models. The human might believe a certain action is impossible, or they might be unaware of a specific constraint the robot must follow. Model Reconciliation is the process of identifying these discrepancies and providing explanations that align the human’s mental model with the agent’s actual model.
The Problem of Mental Model Mismatch
In traditional automated planning, an agent finds a sequence of actions to reach a goal based on its internal model ($M_R$). However, in human-aware AI, we must also consider the human's model of the agent ($M_H$). As Sarath Sreedharan, Tathagata Chakraborti, and Subbarao Kambhampati argue in The Emerging Landscape of Explainable AI Planning and Decision Making, a plan that is optimal in $M_R$ might appear suboptimal or even invalid in $M_H$.
An explanation, in this context, is not just a trace of the agent's logic. Instead, it is a set of updates to the human's model—clarifying capabilities, costs, or constraints—so that the agent's plan becomes "optimal" in the eyes of the human.
Contrastive Explanations and Foils
Humans rarely ask for a full technical breakdown; instead, they ask, "Why did you do A instead of B?" The alternative B is known as a foil.
Benjamin Krarup and Daniele Magazzeni demonstrate in Contrastive Explanations of Plans through Model Restrictions that by treating the foil as a set of constraints, the agent can pinpoint exactly why the human's suggested alternative is inferior. This allows the system to generate a contrastive explanation that specifically addresses the user's misconception. Systems like RADAR-X turn this into an interactive loop, using these foils to elicit user preferences and suggest better plans.
Bridging the Knowledge Gap
A major hurdle in this field is that the agent often does not know what the human believes. Most early frameworks assumed the human's model $M_H$ was known, but recent research focuses on inferring this latent state:
- Model-Free Approaches: Sarath Sreedharan and Alberto Olmo proposed Model-Free Model Reconciliation, which learns to predict user confusion and provides minimal explanations without requiring a formal map of the user's mind.
- Probabilistic Logic: Yinxu Tang and Stylianos Loukas Vasileiou extended this to uncertain environments in Model Reconciliation via Cost-Optimal Explanations in Probabilistic Logic Programming, allowing agents to reconcile beliefs even when information is noisy.
- Dialectical Interaction: Rather than a one-way data dump, DR-Arg uses structured argumentation to resolve discrepancies through a two-way dialogue, making the alignment process more natural for the human.
From Explanation to Alignment
This research direction ultimately merges into the broader "Value Alignment" problem. In Expectation Alignment (EAL), Malek Mechergui and Sarath Sreedharan treat rewards as noisy communications of behavioral expectations. By aligning these expectations through interaction, the agent ensures it isn't just following a literal reward function, but is actually doing what the human intended.