Introduction
In large-scale automated planning environments, such as military air operations, planners rely on complex hierarchical models of entities and processes. Even minor inaccuracies in these models—for instance, an incorrectly set fuel burn rate—can lead to suboptimal plans. While catastrophic failures are easily noticed, they are often too complex to diagnose effectively. This paper proposes that "non-critical anomalies," where a goal is met but behavior deviates slightly from the plan, provide the best opportunity for model refinement. The challenge lies in automating the detection and diagnosis of these subtle discrepancies across thousands of concurrent missions.
System Architecture and Methodology
The authors introduce a multi-agent architecture centered on a Diagnosis Manager and a Mixed-Initiative Model Adaptation loop. This system automates the detection of "expectation failures" and suggests repairs to human experts.
Diagnosis Manager
The Diagnosis Manager handles the lifecycle of an anomaly through several stages:
- Hypothesis Generation: Potential causes are triggered by anomaly arrival, goal failures, or general "sanity check" initializations. The system can move up or down the model hierarchy (e.g., from a specific aircraft to an entire class) to find the right level of abstraction for the error.
- Specialized Agents: Individual Java-based agents are responsible for specific hypotheses. They perform tasks such as statistical analysis of historical data or temporal link analysis to determine if a current discrepancy originated in a previous mission phase.
- Evidence Combination: Using Dempster-Shafer theory, the system merges beliefs from various agents into a single likelihood value for each hypothesis.
Mixed-Initiative Adaptation
Rather than autonomous repair, the system presents high-likelihood suggestions to a user. The user can accept, reject, or ignore these suggestions, and this feedback is used to refine future diagnostic processes. This ensures that human expertise remains central to the model maintenance process.
Experimental Evaluation
The system was evaluated using three artificial datasets designed to mimic real-world fuel level anomalies in military aviation:
- Data Set 1 (Model Error): The model's fuel burn rate is set too high, causing a growing divergence between expected and actual levels.
- Data Set 2 (Sensor Error): A faulty sensor reports zero fuel, creating a "false anomaly" despite the plan being executed correctly.
- Data Set 3 (External Fault): A fuel leak leads to a crash, resulting in both fuel and spatial (altitude) anomalies.
Key Results
The agent-based approach demonstrated significant robustness compared to traditional hand-coded rules. In the case of sensor errors (Data Set 2), the agent-based system correctly identified the "faulty sensor" hypothesis because the data did not match known model-error patterns, whereas naive rules incorrectly flagged a model failure. Furthermore, in complex scenarios like the crash in Data Set 3, the mixed-initiative component allowed human operators to filter out secondary symptoms (like altitude drops) and focus on the root cause (the fuel leak) suggested by the agents.
Conclusion
By treating subtle anomalies as learning signals rather than mere noise, the proposed system provides a scalable way to maintain accurate domain models. The combination of hierarchical hypothesis generation, temporal analysis, and a human-in-the-loop adaptation process allows for effective blame assignment in complex, high-tempo planning environments. Future work will focus on better managing user distraction during complex failures and automating the learning process from user feedback.
Summary updated