Designing Comprehensible Agents

Phoebe SengersIJCAI 1999

Abstract: In this seminal 1998 work, Phoebe Sengers proposes a fundamental shift in AI transparency. Rather than treating agents as "black boxes" that require post-hoc explanation modules, Sengers argues for building agents that are inherently comprehensible. By leveraging principles of narrative psychology, the paper introduces the Expressivator architecture, which structures agent behavior into a coherent story for the human observer.

The Problem: The AI "Black Box"

As AI agents became more prevalent in the late 1990s—appearing in tutoring systems, expert platforms, and entertainment—a significant hurdle emerged: users often found their behavior erratic or inexplicable. Traditional AI design focused on internal problem-solving logic, which rarely aligned with human intuition. When an agent abruptly switched tasks because an internal variable reached a threshold, the transition appeared jarring.

Sengers argues that the common solution of the time—adding a separate module to translate internal states into text—was insufficient. If the underlying architecture isn't designed to be "read" by a human, no amount of secondary explanation can make the behavior feel natural or logical.

The Expressivator Architecture

The core of Sengers' proposal is the Expressivator, an architecture built on the behavior-based language Hap. It applies narrative psychology—the study of how humans use stories to make sense of the world—to the technical design of agents. The system relies on several key innovations:

  • Signs and Signifiers: Behaviors are structured as "signs" intended for communication rather than just internal logic.
  • Sign Management: The system tracks what the user has likely perceived, adjusting the timing and presentation of actions to ensure the user's interpretation keeps pace with the agent's state.
  • Behavioral Transitions: Instead of abrupt state changes, the Expressivator uses "transition demons" to act out the reasoning behind a change. For example, an agent might glance at a clock before switching tasks to signal why it is moving on.
  • Meta-Level Controls: These allow behaviors to query and modify other behaviors, enabling transition sequences to gracefully shut down one activity while priming the next.

Evaluation: The Industrial Graveyard

The architecture was evaluated qualitatively in a virtual environment called the Industrial Graveyard. The primary character, "The Patient" (a discarded lamp), was programmed to interact with its environment and a "bully" character. The goal was to see if users could understand the lamp's motivations through its actions alone.

Key Results and Insights

The efficacy of the Expressivator was demonstrated by comparing the agent's behavior with and without narrative transitions.

  1. Without Transitions: The agent exhibited robotic "stimulus-response" behavior. When sensing a threat, it would instantly begin a task like exercising. While technically optimal, it was perplexing to observers.
  2. With Transitions: The agent would glance at the bully, look at a schedule, check the time, and then begin exercising frantically. This sequence clearly communicated the logic: the agent is performing the task because it is scheduled and it fears the consequences of non-compliance.
Figure 1: A sequence from the Industrial Graveyard showing 'The Patient' (the lamp) checking its schedule and the clock to communicate its internal reasoning to the observer.

Sengers found that this approach did not lead to a "transition explosion." For a system with 8 high-level behaviors, only 15 transitions were required to make the agent's logic visible, proving the method is both computationally and design-feasible.

Conclusion

The Expressivator demonstrates that for an agent to be comprehensible, the "why" is as important as the "what." By prioritizing external expression over internal efficiency and focusing on the "connective tissue" between actions, designers can create AI that humans can naturally understand through narrative. While the system's success depends on high-quality animation and may be domain-specific, it provides a powerful framework for building believable, transparent social and entertainment AI.

Summary updated

BibTeX
@inproceedings{sengers1999designing,
  author = {Phoebe Sengers},
  title = {{Designing Comprehensible Agents}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1999},
}
Figure