Refining the Knowledge Base of a Diagnostic Expert System: An Application of Failure-Driven Learning

Michael J. PazzaniAAAI 1986

Introduction

In the mid-1980s, the development of expert systems for complex hardware diagnosis faced a significant challenge known as the knowledge acquisition bottleneck. Engineers were forced to choose between two suboptimal approaches: fast but brittle heuristic-based rules, which were difficult to extract from human experts, or highly accurate but computationally expensive model-based simulations.

This paper introduces ACES (Attitude Control Expert System), a framework designed to bridge this gap. ACES utilizes Failure-Driven Learning to automatically refine simple, "naive" diagnostic rules into sophisticated heuristics by learning from its own mistakes. By leveraging deep device models only when a failure occurs, the system achieves the efficiency of a heuristic system with the accuracy of a model-based one.

The ACES Architecture and Methodology

ACES operates by maintaining a dual representation of the system it monitors. It begins with a set of "naive" heuristics—definitional rules that are often too broad for real-world application. For example, a naive rule might state: "If a tachometer reads zero, the reaction wheel is broken."

Figure 1: Functional block diagram of the ACES control loop and hardware components.

When an anomaly is detected in the telemetry, the system follows a specific learning loop:

  1. Hypothesis Generation: A naive heuristic proposes a potential fault.
  2. Verification: ACES consults a device model—a functional description of the hardware—to confirm if the hypothesized fault explains the observed symptoms.
  3. Failure Analysis: If the device model denies the fault (a "hypothesis failure"), the system analyzes the model to find the specific reason for the discrepancy.
  4. Rule Specialization: The system then specializes the original heuristic by adding the missing condition discovered during analysis.

Through this process, ACES identifies three primary reasons for failure: Inconsistent Prediction (the fault should cause symptoms that are absent), Unusual Mode ( a rare but normal operational state explains the behavior), or Unusual Input (the component is receiving erroneous data from elsewhere).

Experimental Methodology

To validate this approach, the study utilized a high-fidelity simulator of the DSCS-III satellite (Defense Satellite Communications System). This simulator generates telemetry tapes reflecting real-world faulty behaviors, which serve as the primary input for the ACES diagnostician.

Figure 2: Telemetry data showing attitude error signals (RMER, YMER, PMER) used for fault detection.

The system processes these error signals to determine if the satellite's attitude control is functioning within normal parameters. When the telemetry deviates from expected values, the learning loop is triggered to either confirm an existing rule or generate a more refined one.

Key Results and Discussion

The authors compared three versions of the system: Naive-ACES (initial rules), Naive+Learning (refined rules), and Expert-ACES (rules hand-coded by human engineers). The results demonstrated significant improvements in both accuracy and computational efficiency.

  • Diagnostic Accuracy: After the learning phase, the "Naive+Learning" system matched the performance of the Expert system. Notably, in cases involving rare operational states—such as when two opposing wheel speeds were equal—the learned rules actually outperformed human-coded rules, which had failed to account for that specific edge case.
  • Computational Efficiency: The refinement of rules led to a drastic reduction in the search space. In a test case involving a tachometer fault, the number of logical inferences required for diagnosis dropped from 2,268 to just 211.
  • Hypothesis Reduction: The naive system initially generated 21 false leads for a single fault; the learned system was able to narrow the diagnosis down to the correct fault immediately.

Conclusion

ACES demonstrates the power of Explanation-Based Learning (EBL) in a diagnostic context. Unlike inductive machine learning, which requires large datasets to identify patterns, ACES can learn a general, accurate rule from a single failure because it possesses a physical model to explain the error. By using deep models as "teachers" rather than primary diagnosticians, ACES overcomes the knowledge acquisition bottleneck, creating a system that is both fast enough for real-time monitoring and accurate enough for complex satellite maintenance.

Summary updated

BibTeX
@inproceedings{pazzani1986refining,
  author = {Michael J. Pazzani},
  title = {{Refining the Knowledge Base of a Diagnostic Expert System: An Application of Failure-Driven Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1986},
}
Figure