When an autonomous agent plans its actions, it typically optimizes for a well-defined mathematical objective: minimizing path length, reducing energy consumption, or maximizing expected reward. However, when these agents deploy into environments shared with humans, a striking problem emerges: mathematically optimal plans often look bizarre, unpredictable, or confusing to human observers.
An agent optimizing purely for internal efficiency might make sudden stops, choose counter-intuitive routes, or execute a sequence of actions that leaves human teammates guessing about its ultimate goals. To resolve this, a major direction in human-aware AI focuses on a fundamental trade-off: autonomous systems must intentionally sacrifice physical or computational optimality to achieve behavioral clarity.
The Cost of Pure Efficiency
The tension between machine efficiency and human comprehension is not a new discovery. In early work on agent architecture, Phoebe Sengers argued in Designing Comprehensible Agents that streamlining logic purely for machine execution strips away the expressive signs and transitional steps necessary for a human observer to construct a coherent narrative of the agent's behavior. When systems ignore this narrative, they risk rejection. As Max Horstmann et al. demonstrated, accepting a marginal reduction in mathematical optimality can yield massive gains in the simplicity and auditability of automated plans, which is essential for human trust in high-stakes environments.
To design agents that can navigate this trade-off, researchers categorize observer-aware behavior into three distinct communicative mechanisms:
- Legibility: Acting to reveal an intention as early as possible.
- Explicability: Acting according to what the human expects, given their current understanding of the task.
- Explanation: Explicitly communicating to update the human’s understanding when a sub-optimal or surprising action is necessary.
[ Agent Trajectory ]
|
+-----------------+-----------------+
| |
[ Explicable Action ] [ Optimal Action ]
Matches human mental model. Conflicts with human expectations.
Requires NO explanation. Requires explicit communication.
(Higher execution cost) (Higher cognitive/explanation cost)
Legibility vs. Explicability
While legibility and explicability both require an agent to deviate from its shortest path, they operate on different cognitive mechanisms.
Legibility focuses on intent signaling. Imagine a robot moving toward two closely spaced doors. The shortest path might keep the robot exactly between both doors until the final moment. However, this path is highly ambiguous to an observer. A legible planner will intentionally curve its trajectory early toward the target door. Salomé Lepers et al. showed that agents optimized for predictability select longer, more constrained paths because they provide clearer signals to external observers. In team settings, Yanyu Liu et al. proved that choosing these sub-optimal individual trajectories allows teammates to synchronize their actions much earlier, ultimately reducing the team's total time to task completion.
Explicability, by contrast, is about conforming to expectations. If a human believes a specific hallway is blocked, they expect a robot to take an alternative route. An explicable robot will take the longer alternative route—even if its superior sensors detect that the hallway is actually clear—simply to avoid surprising the human. As outlined in the overview of human-aware AI systems (Challenges of Human-Aware AI Systems), choosing an explicable plan allows the agent to bypass the cognitive overhead and temporal cost of having to explain its actions.
Formalizing the Observer's Mind: OAMDPs
To compute these behaviors, an agent cannot simply plan within its own physical state space. It must treat the observer's mind as part of the environment.
Shuwa Miura et al. formalized this through the Observer-Aware Markov Decision Process (OAMDP). In a standard MDP, the state space consists of physical variables (e.g., coordinates on a grid). In an OAMDP, the state space is augmented to include the observer's internal belief state:
$$S_{\text{OAMDP}} = S_{\text{physical}} \times B_{\text{observer}}$$
When the agent takes an action, the environment transitions physically, and the observer updates their belief $b \in B_{\text{observer}}$ via a cognitive model (such as Bayesian inference). The reward function explicitly balances task performance against the clarity of the action:
$$R(s, b, a) = R_{\text{task}}(s, a) + \alpha \cdot R_{\text{clarity}}(b)$$
By tuning $\alpha$, the planner can systematically trade physical optimality (like travel time or energy) to increase the speed at which an observer correctly infers the agent's goals. This framework adapts to complex real-world constraints; for instance, Salomé Lepers et al. extended this to partial observability (PO-OAMDPs), demonstrating that agents will intentionally choose longer, highly visible paths to maintain predictability rather than taking shorter routes through areas hidden from the observer's field of view.
The Unified Frontier: Explicability and Explanation
What happens when the optimal path is so much more efficient than the expected path that taking the explicable route causes massive performance degradation? In these scenarios, the agent must switch from acting explicably to acting optimally and explaining why.
Tathagata Chakraborti et al. unified these concepts into a single optimization problem known as the MEGA framework. They proved that explicability and explanation are not separate tasks, but rather two sides of the same coin: explicability is simply a special case of an explanation with zero length.
When an agent's internal model $M^R$ differs from the human's mental model of the agent $M^H$, the agent faces the Model Reconciliation Problem (MRP). As formulated by Tathagata Chakraborti et al., instead of generating a long, confusing justification based on its own complex logic, the agent identifies a minimal "patch" of information $\mathcal{E}$ to share with the human. Once the human applies this patch to their mental model, the agent's seemingly bizarre action suddenly appears optimal to them.
[Human Model M^H] + [Explanation Patch E] ---> [Updated Human Model M^H*]
|
Agent's optimal plan now
makes sense to the human.
By minimizing a weighted sum of the plan's execution cost and the length of the explanation patch, the agent dynamically decides the most efficient mode of interaction (Tathagata Chakraborti et al.). To prevent information overload, Sarath Sreedharan et al. developed model-free methods to compute these minimal explanatory messages without requiring a complete, declarative map of the user's mind.
Simplicity as a First-Class Objective
This trade-off extends beyond single-agent path planning into multi-agent coordination and policy representation. In multi-agent path finding, the most direct routes often involve intricate, overlapping trajectories. Justin Kottinger et al. showed that introducing artificial delays or longer routes simplifies the visual complexity of agent interactions, making the global plan significantly easier for a human supervisor to verify and trust.
Ultimately, this line of research aligns with deep-seated human cognitive biases. Carlos G. Correa et al. discovered that human planning strongly prioritizes structural simplicity and subroutine reuse over raw action minimization. This preference directly impacts human-AI collaboration: Raphaël Koster et al. observed that human participants frequently prefer distilled, simplified policy formulas over highly efficient but complex black-box algorithmic agents. A simpler, slightly sub-optimal strategy is perceived as fairer and more predictable, proving that in human environments, legibility is often more valuable than mathematical perfection.
Key Foundational Work
Automated Generation of Understandable Contingency Plans (2003) — contributed 2 concept elements to this direction.
Designing Comprehensible Agents (1999) — contributed 2 concept elements to this direction.