Flexibly Exploiting Prior Knowledge in Empirical Learning

Julio Ortega, Douglas Fisher

Introduction

In the field of artificial intelligence, a long-standing challenge involves effectively integrating expert domain theories with empirical data. Traditional approaches often forced a choice between rigid expert systems and purely data-driven inductive models. This paper presents a flexible framework for biasing decision tree learning toward features derived from expert models, allowing for a graceful transition from prior knowledge to empirical evidence as data availability increases.

The Problem of Dogmatic Learning

Mid-90s machine learning systems often suffered from "dogmatism." Expert systems relied on human knowledge that could be shallow or incorrect, while inductive learners required large amounts of representative data to avoid overfitting to noise. Previous hybrid systems were either too rigid—following expert rules until explicitly contradicted—or too eager to revise theories based on potentially noisy data. The authors sought a middle ground: a system that defaults to expert knowledge when data is scarce but can be "convinced" by statistically overwhelming empirical evidence to pivot toward raw data patterns.

Methodology: Hierarchical Feature Augmentation

The proposed method modifies the C4.5 decision tree algorithm through three primary mechanisms:

  1. Feature Augmentation: Raw data is processed through an expert model to generate "intermediate terms" as additional features.
  2. Preference Hierarchy: Features are categorized into levels based on their proximity to the expert's final conclusion. Level 0 represents the model's final prediction, Levels 1 and 2 represent intermediate concepts and rules, and Level 3 consists of raw, observable data.
Figure 1: The preference hierarchy used to categorize features from model predictions (Level 0) down to raw data (Level 3).
  1. Statistical Bias via Bootstrapping: Unlike standard C4.5, which selects features solely based on information gain, this algorithm prefers features at lower levels (closer to expert knowledge). A higher-level (raw) feature is only selected if it is statistically significantly better at predicting the outcome than the preferred lower-level feature. This significance is determined using Bootstrap Hypothesis Testing, resampling the training data to ensure that any observed empirical superiority is robust and not a result of noise.

Experimental Evaluation

The system was evaluated using the UCI Audiology dataset (226 examples, 24 classes). To test robustness, the researchers introduced "imperfect models" (corrupted by deleting 5% to 50% of rules) and "skewed data" (unrepresentative training sets).

Key Results and Insights

The biased learner demonstrated several advantages over standard inductive methods:

  • Data Efficiency: High accuracy was achieved with significantly fewer examples. With a perfect model, near-perfect accuracy was reached with only ~80 examples, a threshold where standard C4.5 struggled.
  • Robustness to Noise and Skew: Model-derived features acted as a buffer against "jittery" raw data. The system proved resilient even when the provided expert model was mediocre (e.g., 65% accuracy), using it as a starting point and correcting it only where empirical evidence was strongest.
  • Dynamic Adaptation: As shown in the experimental results, the system maintains high-level feature usage when data is thin but shifts toward lower-level features as the training set size grows and empirical patterns become more reliable.
Figure 4: Accuracy performance (top) and the shift in root feature level (bottom) as a function of training set size across various levels of model corruption.

Conclusion

By treating expert knowledge as a flexible bias rather than an immutable truth, this approach allows machine learning models to exploit the strengths of both deduction and induction. While bootstrapping introduces computational overhead, the gains in accuracy and robustness—especially in data-poor or noisy environments—provide a compelling case for hybrid learning systems. Future work aims to automate the selection of significance levels and extend the framework to more complex domain theories.

Summary updated

BibTeX
@inproceedings{ortega1995flexibly,
  author = {Julio Ortega and Douglas Fisher},
  title = {{Flexibly Exploiting Prior Knowledge in Empirical Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
}

Loading related content...

Figure