An Information Theoretic Approach to Rule-Based Connectionist Expert Systems

Rodney M. Goodman, John W. Miller, Padhraic SmythNeurIPS 1988 · OpenReview

Overview

This paper, presented by researchers from Caltech and NASA’s Jet Propulsion Laboratory, explores a hybrid approach to artificial intelligence that bridges the gap between symbolic AI (interpretable rules) and connectionism (parallel neural networks). By leveraging the rigorous framework of information theory, the authors present a method to automatically extract probabilistic "if-then" rules from raw data and map them into a neural network. The resulting expert system is both computationally efficient and human-readable, avoiding the notorious "black box" problem of traditional neural networks.

The Knowledge Engineering Bottleneck

In the late 1980s, AI faced a significant divide. Expert systems excelled at reasoning but required tedious manual labor to encode knowledge from human experts—a hurdle known as the knowledge engineering bottleneck. Conversely, connectionist models could learn from data automatically but operated as opaque "black boxes," making it impossible to audit their decision-making processes. Furthermore, traditional expert systems struggled to handle "fuzzy" or uncertain data gracefully.

The authors address these issues by automating rule discovery and ensuring that the weights in their resulting neural network have clear, probabilistic semantics.

Methodology: From Data to Networks

The researchers propose a two-step pipeline that transforms raw data into a functioning, interpretable neural network.

1. Learning via the ITRULE Algorithm

Instead of relying on standard backpropagation, the system uses an algorithm called ITRULE. This algorithm searches a database for the most "informative" rules based on a metric called the J-measure, derived from Shannon’s information theory. The J-measure does not merely evaluate how often a rule is correct; it weights the "information gain" of a rule by the probability that the rule will be triggered in the real world. This naturally balances a rule's simplicity (generality) with its predictive power, preventing the system from overfitting to rare anomalies.

Top 10 rules extracted by ITRULE for the mutual fund dataset, ranked by J-measure.

2. Inference via the Rule Network

Once the $K$ best rules are identified, they are mapped onto a two-layer recurrent neural network. In this architecture, nodes represent specific propositions (e.g., "The fund is high-risk"). Unlike standard neural networks where weights are arbitrary values tuned via gradient descent, the weights here are explicitly set to specific information-theoretic values, such as log-odds ratios or J-measure values.

The Rule Network architecture mapping extracted rules into a two-layer recurrent neural network.

The authors propose two specific network architectures for inference: the Hypothesis Testing Network (based on Neyman-Pearson decision theory) and the Uncertainty Network (based on bit-rate changes in probability distributions).

Experimental Validation

The approach was validated using a financial database of 88 no-load mutual funds. The dataset included both categorical attributes (e.g., fund type) and numerical data (e.g., 5-year return, expense ratio). To be compatible with the rule-based neurons, continuous numerical data was first quantized into discrete binary values (e.g., "Return > S&P 500" vs. "Return < S&P 500").

Key Results

The researchers compared their two network models against the theoretical Bayes' risk, which represents the minimum possible error rate for a given dataset.

Units Clamped (Knowns) Uncertainty Test Accuracy Hypothesis Test Accuracy 1 - Bayes' Risk (Optimal)
9 66.8% 70.4% 88.6%
5 70.1% 70.1% 80.6%
2 48.2% 63.0% 63.6%
1 51.4% 65.7% 64.8%

The Hypothesis Testing Network consistently outperformed the Uncertainty Network. Remarkably, the network demonstrated highly robust inference: when only a few attributes were known (1 or 2 "clamped" units), the network's performance was nearly identical to the theoretical mathematical optimum.

Limitations and Future Directions

While promising, the authors note several limitations:

  • Independence Assumptions: The current models assume that different pieces of evidence are conditionally independent. Acknowledging this simplification, the authors suggest that incorporating "higher-order" rules (combinations of three or more variables) could improve accuracy.
  • Data Discretization: The system requires continuous data to be quantized into discrete categories, which can result in a loss of nuance.
  • Scale: Although validated on a small dataset of 88 samples, the ITRULE approach is ultimately designed to scale to much larger databases in complex domains like telecommunications and medicine.

Summary updated

Abstract

We discuss in this paper architectures for executing probabilistic rule-bases in a par(cid:173) allel manner, using as a theoretical basis recently introduced information-theoretic models. We will begin by describing our (non-neural) learning algorithm and theory of quantitative rule modelling, followed by a discussion on the exact nature of two particular models. Finally we work through an example of our approach, going from database to rules to inference network, and compare the network's performance with the theoretical limits for specific problems.
BibTeX
@inproceedings{goodman1988information,
  author = {Rodney M. Goodman and John W. Miller and Padhraic Smyth},
  title = {{An Information Theoretic Approach to Rule-Based Connectionist Expert Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1988},
  url = {https://openreview.net/forum?id=Sk--DvW_Zr},
}
Figure