In the early 1990s, as artificial intelligence agents grew in complexity—integrating perception, planning, and real-time reaction—the ability to explain their behavior became a significant hurdle. This 1994 paper from the AAAI proceedings introduces Debrief, a system designed to help intelligent agents explain the "why" behind their actions. Rather than providing a raw log of code or rules, Debrief enables an agent to use "mental simulation" to reconstruct past decisions, experiment with alternative scenarios, and learn the underlying rationales for its behavior.
The Problem: The Unreadability of Rule Traces
Traditional methods for explaining AI behavior typically relied on "rule traces," which are essentially printouts of every if-then statement that fired during a process. For human users, these traces are often unreadable, cluttered with implementation details, and lacking high-level "domain rationale."
In high-stakes environments like military flight simulations (TacAir-Soar), human trainees and evaluators need to know if an agent is behaving correctly for the right reasons. For example, a pilot needs to know "I fired because the target was hostile and within range," rather than "Rule #402 fired because variable X was 1." Debrief was developed to bridge this gap between low-level execution and high-level justification.
The Debrief Architecture: Learning via Mental Simulation
Built on the Soar cognitive architecture, Debrief employs a "detective work" loop to uncover the causes of its own actions. The process involves several distinct stages:
- Episodic Memory & Recall: The agent records state changes during a mission. When queried, it uses a "Recall-State" process to reconstruct the exact world state at the moment of a specific decision.
- Mental Simulation: The agent re-runs its decision-making logic in a "sandbox" version of that recalled state.
- Feature Ablation (The "What-If" Test): To identify what truly mattered, Debrief systematically perturbs situational factors (e.g., "What if the bogey wasn't hostile?"). If the agent’s decision changes, that attribute is flagged as a "critical factor."
- Learning via Chunking: Soar uses a mechanism called "chunking" to cache the results of these simulations. Over time, the agent builds an abstract model of its own behavior, allowing it to explain similar future actions instantly without re-simulating the entire decision process.
Generating High-Level Rationales
By using mental simulation to find critical factors, the system automatically filters out "implementation noise." Regardless of how many hundreds of rules fired, if the decision only changes when the "target distance" variable is modified, then "target distance" is the primary component of the explanation. This allows the system to generate structured, readable rationales that address both actions and the beliefs that led to them.
Evaluation and Key Insights
The system was implemented and tested within TacAir-Soar, a high-fidelity air-to-air combat simulator. Formative evaluations were conducted with US Naval Reserve fighter pilots, who interacted with the agent and provided feedback on the relevance of its explanations.
- Functional Scope: Debrief successfully handled 66 different types of events, ranging from situation assessments to kinetic actions like firing missiles.
- Efficiency: In a typical session, the system generated between 500 and 1,000 "chunks" (learned rules), providing significant speedups as the agent reused rationales across different missions.
- Self-Modeling: A profound takeaway from this research is that the agent is not just modeling the world; it is learning to model itself. It treats its own decision-making process as an external system to be studied and summarized.
Limitations and Future Work
Despite its success, the 1994 version of Debrief faced challenges. It struggled to explain decisions based on the absence of a feature (e.g., "I didn't turn because I didn't see an enemy") due to limitations in Soar's chunking mechanism at the time. Additionally, the natural language generation was limited to a small vocabulary and simple templates.
However, the authors noted that this "mental simulation" framework could eventually be used for more than just explanation. It could serve as a basis for self-correction, allowing agents to discover better tactical choices by "dreaming" about alternatives during downtime, effectively learning from hypothetical scenarios as well as real ones.
Summary updated