When an AI agent plans a sequence of actions, it optimizes against its own internal mathematical model of the world. We often assume that human observers share this exact same model. However, as Sarath Sreedharan et al. point out, discrepancies in knowledge, capabilities, and abstraction levels are the norm, not the exception.
When these models diverge, a breakdown in coordination occurs. An action that is perfectly optimal under the AI's internal model can appear completely irrational, inefficient, or erroneous to a human observer (Sarath Sreedharan). Traditional Explainable AI (XAI) often treats explanation as a soliloquy—the agent simply outputs its internal logs or decision-making traces. But if the human's underlying assumptions are different, a raw dump of the AI's internal logic will only increase cognitive load and confusion, a phenomenon observed even in expert domains like clinical medicine (Angus Nicolson et al.).
Explanation as a Minimal Model Update
To solve this, Tathagata Chakraborti et al. formalized the Model Reconciliation Problem (MRP). Instead of justifying an action in isolation, the AI treats an explanation as a targeted update to the human's mental model.
To understand why this works, imagine two distinct representations:
- The robot's true model ($M^R$), which contains the real environment dynamics and constraints.
- The human's mental model of the robot ($M^H$), which represents what the human thinks the robot can or should do.
[ Robot Model M^R ] [ Human Model M^H ]
│ │
Computes optimal plan π* Expects alternative plan
│ │
└───────────► [ Mismatch ] ◄───────────┘
│
Find minimal update (E)
│
[ M^H + E = π* is optimal ]
The agent computes an optimal plan $\pi^*$ based on $M^R$. To the human, $\pi^*$ looks suboptimal because they evaluate it using $M^H$. The goal of model reconciliation is to find a minimal set of modifications $\mathcal{E}$ (such as revealing a hidden precondition or an unexpected action effect) such that when the human updates their model to $M^H + \mathcal{E}$, the agent's plan $\pi^*$ suddenly becomes optimal and rational in their eyes.
By focusing only on the mismatches between $M^R$ and $M^H$, the agent minimizes the information it needs to communicate. This makes explanations highly efficient, resolving user confusion with minimal cognitive overhead (Stylianos Loukas Vasileiou et al.).
The Trade-off Between Acting and Explaining
An agent facing a misaligned human model has two choices: it can change its behavior to match human expectations, or it can use communication to change those expectations. The authors formalized this as a trade-off between explicability and explanation via the MEGA algorithm (Tathagata Chakraborti et al.).
- Explicability requires the agent to sacrifice its own optimality to perform actions that make sense within the human's flawed model $M^H$. For example, a robot might take a longer route simply because it knows the shorter route will confuse an observer who doesn't know the path is clear.
- Explanation allows the agent to act optimally according to $M^R$ and then provide a verbal or visual reconciliation to correct the human's misconceptions.
By balancing these two modalities, an autonomous system can choose the path of least resistance—sometimes taking a slightly suboptimal route to avoid confusing a user, and other times executing the true optimal path and briefly explaining why (Tathagata Chakraborti et al.).
Answering the Contrastive Question
Humans rarely ask for an explanation in a vacuum; they usually ask contrastive questions: "Why did you choose plan A instead of my expected plan B?"
Benjamin Krarup et al. showed how to operationalize this by compiling the human's preferred alternative (the "foil") directly into the planning problem as a set of model restrictions. By forcing the planner to try and execute the human's expected plan, the system can pinpoint exactly why that alternative fails or is highly inefficient under the true model $M^R$. The explanation then directly addresses the foil (e.g., "I couldn't do B because precondition X was not met"), directly resolving the mental model mismatch.
Algorithmic Foundations and Scaling Up
How do agents computationally search for these minimal model updates? Early methods relied on expensive planning loops, but recent breakthroughs treat model reconciliation as a pure knowledge representation and logic problem.
Stylianos Loukas Vasileiou demonstrated that by representing both the agent and human models as sets of logical formulas, finding the minimal explanation maps directly to the mathematical duality between hitting sets and logical inconsistencies. This formulation allows logic solvers to find optimal explanations orders of magnitude faster than traditional planning-based search.
Furthermore, this logic-based perspective allows model reconciliation to scale beyond classical deterministic planning into complex hybrid systems with continuous processes (Stylianos Loukas Vasileiou et al.) and noisy, probabilistic environments (Stylianos Loukas Vasileiou et al.).
Overcoming the Need for Perfect Mental Maps
A common critique of early model reconciliation frameworks was their reliance on a known, explicit map of the human's mind ($M^H$). In practice, humans are dynamic, biased, and hard to model explicitly (Yinxu Tang et al.). If the agent's model of the human is wrong, its explanations fail to hit the mark.
To bypass this bottleneck, Sarath Sreedharan et al. introduced a model-free approach. Instead of maintaining a declarative representation of human beliefs, the agent learns a predictive model of user confusion directly from data. It can then identify the minimal set of explanatory messages needed to resolve that confusion without ever needing an explicit map of the user's internal logic.
Ultimately, this shift highlights that value alignment is fundamentally constrained by concept alignment (Sunayana Rane et al.). Whether implemented through explicit logical hitting sets or data-driven confusion predictors, the core insight remains unchanged: effective explanation is not about showing how the AI processes its internal logic, but about surgically repairing the rift between the machine's reality and the human's expectations.