When an autonomous agent—whether a warehouse robot or a software assistant—calculates the "optimal" way to complete a task, it usually minimizes a mathematical cost: time, energy, or compute. However, these mathematically perfect paths often appear erratic, confusing, or even alarming to human observers. A robot that suddenly stops to recalculate or takes a counter-intuitive shortcut may be efficient, but it is not legible.
Research in human-aware planning suggests that for AI to be a collaborative partner, it must often sacrifice raw efficiency to remain understandable. This creates a fundamental "efficiency-legibility tradeoff": the cost of being clear versus the cost of being fast.
The Gap Between Two Models
The core of the problem lies in a mismatch of mental models. As Subbarao Kambhampati argues in Challenges of Human-Aware AI Systems, an agent operates based on its internal model ($M_R$), while the human observer expects the agent to act according to their own mental model of the task ($M_H$).
When $M_R$ and $M_H$ diverge—perhaps because the robot knows a hallway is blocked but the human doesn't—the robot's optimal action will seem inexplicable. Phoebe Sengers noted as early as 1999 in Designing Comprehensible Agents that agents should be designed for comprehensibility from the start, rather than treating explanation as an afterthought. She argued that logic streamlined for machines often omits the "expressive signs" humans need to construct a coherent narrative of the agent's goals.
Balancing Performance and Clarity
To bridge this gap, researchers have formalized the Observer-Aware MDP (OAMDP). In A unifying framework for observer-aware planning and its complexity, Shuwa Miura and Shlomo Zilberstein describe a mathematical framework where agents explicitly balance task performance against the speed at which an observer can infer their goals.
This balance typically manifests in three ways:
- Legibility: The agent chooses actions that make its destination clear as early as possible.
- Predictability: The agent chooses actions that make its next move easy to guess.
- Explicability: The agent chooses actions that align with what the human expects it to do, even if those actions are sub-optimal.
In Explicable Robot Planning as Minimizing Distance from Expected Behavior, Anagha Kulkarni and colleagues show that agents can learn to map mathematical plan distances to human "explicability scores." This allows a robot to choose a slightly longer path that requires no explanation over a shorter path that would baffle a supervisor.
Model Reconciliation: To Act or To Explain?
When the cost of acting explicably becomes too high—for instance, if the "expected" path is blocked or dangerously inefficient—the agent faces a second tradeoff: act or explain.
Tathagata Chakraborti and Subbarao Kambhampati frame this as Model Reconciliation. In Plan Explanations as Model Reconciliation, they demonstrate that a robot can maintain trust by providing the minimum set of model updates (e.g., "The bridge is out") to make its seemingly "weird" optimal plan appear optimal to the human too.
The decision depends on the "efficiency tax." As Yu Zhang notes in Plan Explainability and Predictability for Cobots, a small decrease in efficiency often leads to a disproportionately large gain in human trust. However, if the efficiency loss is too great, the agent should instead invest in the cognitive overhead of a verbal explanation.
Trust as a Dynamic Resource
This tradeoff isn't static; it evolves. Mudit Verma and Zahra Zahedi propose in Trust-Aware Planning that trust is a resource to be managed. A robot might strategically choose explicable (but sub-optimal) actions early in an interaction to build a "trust buffer," which it can later "spend" by performing highly efficient but surprising actions without losing the user's confidence.
Ultimately, this research direction moves AI away from "black-box" optimization toward a "socially aware" optimization. Whether through sparse contrastive explanations or inspectable tools, the goal is the same: an agent whose actions are not just correct, but clearly and obviously correct to the people working alongside them.