Knowledge Base Refinement Using Apprenticeship Learning Techniques

David C. WilkinsAAAI 1988

Introduction

In the late 1980s, the development of expert systems—AI designed to emulate human specialists—faced a significant bottleneck. Refining these systems typically relied on "induction from examples," a process where models were fed large datasets of symptoms and final diagnoses to identify patterns. However, this approach often overlooked the nuanced reasoning process experts use to reach a conclusion.

This paper introduces ODYSSEUS, an apprenticeship learning system that refines knowledge bases by "watching" human experts work. Unlike traditional machine learning that focuses on bulk data induction, ODYSSEUS identifies specific gaps in a system's logic by attempting to explain every move a human expert makes. By identifying "explanation failures," the system can conjecture and test new rules to complete its understanding of the expert's strategy.

The ODYSSEUS System

ODYSSEUS operates within the HERACLES expert system shell. Its primary innovation is the concept of Learning by Completing Failed Explanations. The system monitors a human expert (e.g., a physician) as they perform a task, such as asking specific diagnostic questions.

Figure 1: The ODYSSEUS learning cycle, illustrating the process of observing human actions, detecting knowledge base defects, and suggesting repairs.

The learning process follows a structured cycle as shown in the flowchart above:

  1. Observation: The system records an expert's action, such as a "findout" question regarding a patient's symptoms.
  2. Explanation: ODYSSEUS attempts to construct a logical proof justifying why the expert took that specific action based on the current state of the case and the system's internal goals.
  3. Detection: If no logical path can be found to justify the action, an "explanation failure" is triggered, indicating a gap in the domain knowledge.
  4. Repair: The system backchains from the goal to the action to identify the "missing link"—a rule or fact that would make the expert's behavior logical.
  5. Confirmation: Proposed rules are vetted using a "confirmation theory" to ensure they are medically plausible and statistically sound across other cases.

Experimental Setup and Datasets

The researchers evaluated ODYSSEUS using the NEOMYCIN knowledge base, which is designed for diagnosing neurological disorders. The system was tested against a validation set of 112 actual patient cases from Stanford Medical Hospital.

Remarkably, the training set consisted of only two problem-solving protocols recorded from a human physician. This highlights the efficiency of apprenticeship learning compared to traditional inductive methods that require thousands of examples to achieve similar refinements.

Results and Discussion

The impact of observing just two cases was substantial. Before the learning phase, the NEOMYCIN system correctly diagnosed 31% of the test cases. After ODYSSEUS integrated the knowledge gleaned from the expert's process, the accuracy rose to 44%.

Figure 4: Performance of the NEOMYCIN system after learning from two cases, showing a total of 49 correct diagnoses out of 112 cases (approximately 44%).

Key insights from the study include:

  • Process Over Outcome: ODYSSEUS learns from the individual steps of a diagnosis. Even if an expert reaches an incorrect final conclusion, their intermediate questions often reveal valid medical relationships.
  • Strategy vs. Domain Knowledge: The system assumes its high-level diagnostic strategy is correct, using it as a template to identify missing domain-specific facts.
  • Cognitive Economy: The system successfully modeled how experts use general questions to efficiently rule out broad categories of diseases, a behavior that standard induction often fails to capture.

Limitations and Future Work

While effective, ODYSSEUS has several limitations. It assumes its "strategy knowledge" is perfect and cannot learn new reasoning methods. Additionally, the current search for repairs assumes only a single piece of knowledge is missing at any given time; complex gaps requiring multiple simultaneous rules remain a challenge. Future work aims to expand the "confirmation theory" to handle more complex knowledge structures and to address "sociopathic" interactions where new rules conflict with existing ones.

This paper was originally presented at the Seventh National Conference on Artificial Intelligence (AAAI-88).

Summary updated

BibTeX
@inproceedings{wilkins1988knowledge,
  author = {David C. Wilkins},
  title = {{Knowledge Base Refinement Using Apprenticeship Learning Techniques}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1988},
}
Figure