Abstract
In the late 1980s, machine learning was divided between inductive learning from data and explanation-based learning from prior theories. This paper proposes a hybrid approach called Theory Revision, implemented through the Reduced Theory Learning System (RTLS). By "flattening" hierarchical expert rules into operational labels, RTLS can refine imperfect theories using empirical evidence. When tested on the Rheum rheumatology knowledge base, the system significantly improved diagnostic accuracy from 73% to over 90%, demonstrating a robust method for maintaining and improving expert systems.
Introduction
Expert systems rely on complex rule-based theories provided by human specialists. However, these theories are frequently imperfect, leading to false positives or missed diagnoses. Traditional inductive learning often ignores existing knowledge, while pure explanation-based methods struggle with theoretical errors. Allen Ginsberg's 1988 paper introduces a middle path: Theory Revision. The goal is to modify an existing theory to match real-world data while preserving its original language and structure.
Methodology: The RTLS Framework
The proposed system, RTLS (Reduced Theory Learning System), operates through a process of "Prior Operationalization" followed by iterative refinement.
Theory Reduction
The first step involves "flattening" a hierarchical set of rules. The system calculates a label for every hypothesis—a minimal set of "observables" (input data) that trigger that hypothesis. This operationalizes the theory, defining it entirely in terms of observable data before training begins.
Refinement Phases
Once reduced, the theory is refined using a training set of cases through several distinct phases:
- Massive Label Refinement: A statistical approach that adjusts correlation strengths between observables and hypotheses to correct systematic errors.
- Focused Label Refinement: A targeted approach to fix individual false positives (specialization) and false negatives (generalization).
- Solving All Problems: A final pass to ensure 100% accuracy on the training data.
The formal notation used to define these processes is summarized in the table below.
Evaluation and Results
The system was evaluated using the Rheum knowledge base, a medium-sized expert system for rheumatology consisting of 84 observables, 73 hypotheses, and 367 rules. The initial theory reduction produced over 34,000 unique "environments" (combinations of observables).
Performance Metrics
- Accuracy Improvement: The original Rheum theory correctly diagnosed approximately 73% of cases (88 out of 121). After RTLS refinement, accuracy on independent test sets reached 90% to 100%.
- Error Rate: Using leave-one-out cross-validation, the estimated error rate was reduced to 6.7%.
- Statistical Impact: The "Massive Refinement" phase alone contributed a 3-5% performance boost, highlighting the value of combining statistical correlation with case-based logic.
- Efficiency: While the initial reduction was computationally intensive (10 hours), the refinement phase was highly efficient, requiring only 7-10 CPU minutes.
Discussion and Limitations
RTLS demonstrates that "flattening" a theory makes inductive learning more tractable by removing the need to identify specific broken links in a deep rule chain. However, the approach faces scaling challenges; for very large systems, full reduction may lead to a combinatorial explosion of environments. Future work involves refining "partial reduction" strategies and detailing the "retranslation" process—converting modified flat labels back into the original hierarchical rule format to maintain explainability.
Summary updated