Using Qualitative Models to Guide Inductive Learning

Peter Clark, Stan MatwinICML 1993 · OpenReview

This paper, originally presented at the 10th International Machine Learning Conference (1993), addresses a classic tension in artificial intelligence: the gap between "black-box" statistical learning and human-understandable domain knowledge. Authors Peter Clark and Stan Matwin propose a method to use high-level, qualitative models of a system to "guardrail" the learning process, ensuring that the rules discovered by an algorithm are not only accurate but also physically sensible.

TL;DR

By using qualitative models to constrain the search space of a rule-induction algorithm, the authors demonstrate that machine learning can become faster, 100% explainable, and—most surprisingly—more accurate by filtering out "lucky" but nonsensical statistical correlations.

The Problem: Knowledge-Poor Induction

In the early 1990s, inductive learning tools like ID3 or CN2 were gaining traction, but they suffered from a "knowledge-poor" approach. They looked for patterns in data without any understanding of the underlying physics or logic of the domain.

This led to two major issues:

  1. Inexplicability: Algorithms often produced rules that performed well on test data but were "silly" to human experts (e.g., predicting a car's speed based on the color of the driver's shirt if that happened to correlate in the sample).
  2. Efficiency: Without guidance, algorithms had to search a massive "hypothesis space" of possible rules, which was computationally expensive and prone to overfitting.

The authors aimed to bridge this gap by allowing a "Qualitative Model" (QM)—a simplified map of how variables influence one another—to guide the induction process.

Methods: Constraining the Search Space

The researchers modified the CN2 algorithm, a standard tool for inducing "if-then" rules from data. Their approach follows three main steps:

  1. Knowledge Representation: They define a Qualitative Model using nodes (parameters) and arcs (relationships). They use labels like $I+$ (immediate positive influence), $Q+$ (monotonic relationship), and a novel $I*+$ (representing a self-stabilizing feedback loop, like a car reaching a terminal velocity).
Qualitative model of a car's acceleration demonstrating the I*+ relationship.
  1. Rule Extraction: Before looking at the data, the system exhaustively generates all "sensible" rule schemata (templates) that are consistent with the paths in the QM. For example, if the model says "Gas Pedal $\rightarrow$ Engine Power $\rightarrow$ Speed," a rule linking Gas Pedal to Speed is deemed sensible, while a rule linking them inversely is not.
  2. Constrained Induction: During the actual learning phase, the CN2 algorithm's search is restricted. As it explores potential rules, it checks them against the "sensible" lookup table. If a rule doesn't align with the QM, it is discarded immediately.

Datasets and Evaluation

The authors evaluated their method on two simulated process control domains:

  • Water Tank System: A network of seven tanks with feedback loops where the goal is to predict if the water level in the final tank will rise or fall.
The physical layout of the Water Tank System alongside its corresponding qualitative model graph.
  • Ore Grinding Circuit: A complex industrial simulation involving ball mills, screens, and cyclones used in mining. This domain is significantly more "noisy" and interconnected, with 10 observable parameters.

In both cases, numeric simulators were used to generate training and testing sets of varying sizes (from 20 to 215 examples).

Key Insights and Results

  • Constraints Improve Accuracy: Counter-intuitively, restricting the learner's choices actually made it more accurate. By forbidding "impossible" rules, the QM acted as a filter that removed rules which only appeared valid due to random noise in small datasets. In the water tank domain, accuracy increased by up to 4% over the standard algorithm.
  • Quantifying Knowledge: The authors introduced the concept of "example-equivalence." They calculated how many extra training examples a qualitative model is "worth." For the water tank, the QM was worth roughly 0.5 extra examples for every real example provided.
  • 100% Explainability: Because every induced rule is tied to a path in the QM, the system can automatically generate a causal explanation for its predictions (e.g., "I predict Speed will increase because the Gas Pedal was depressed, which increases Engine Power..."). While the standard algorithm produced "sensible" rules only about 40-50% of the time, the QM-constrained version was, by definition, fully explainable.
  • Efficiency Trade-offs: In the water tank system, CPU time was slashed by more than half because the search space was so much smaller. However, in the highly-connected ore circuit, CPU time actually increased slightly due to the computational overhead of checking the complex QM constraints.

Limitations and Future Work

  • Model Quality: The method assumes the human-provided QM is "reasonable." A flat-out wrong model would lead to "negative example-equivalence," actually harming the learner's performance.
  • Thresholds: The QM helps with the structure of the rules but doesn't help find the specific values (e.g., it knows "Temperature" is relevant, but the data must still reveal that "100°C" is the critical threshold).
  • Closing the Loop: The authors suggest a future "reverse process" where, if the data strongly and consistently contradicts the QM, the system could suggest ways to revise the human expert's model.

Summary updated

Abstract

This paper presents a method for using qualitative models to guide inductive learning. Our objectives are to induce rules which are not only accurate but also explainable with respect to the qualitative model, and to reduce learning time by exploiting domain knowledge in the learning process. Such explainability is essential both for practical application of inductive technology, and for integrating the results of learning back into an existing knowledge-base. We apply this method to two process control problems, a water tank network and an ore grinding process used in the mining industry. Surprisingly, in addition to achieving explainability the classificational accuracy of the induced rules is also increased. We show how the value of the qualitative models can be quantified in terms of their equivalence to additional training examples, and finally discuss possible extensions.
BibTeX
@inproceedings{clark1993using,
  author = {Peter Clark and Stan Matwin},
  title = {{Using Qualitative Models to Guide Inductive Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {1993},
  url = {https://openreview.net/forum?id=HJEV8hWu-S},
}
Figure