Combining Symbolic Learning Techniques and Statistical Regression Analysis

Carlo BerzuiniAAAI 1988

This research, originally presented at AAAI-88, explores a hybrid territory between the "black box" of statistical regression and the "white box" of symbolic machine learning. It provides a framework for turning complex, noisy data into interpretable decision rules that domain experts, such as medical professionals, can trust and utilize.

The Interpretability Dilemma

In the late 1980s, AI researchers faced a significant hurdle. Symbolic learning methods (like decision trees) produced rules that were easy for humans to understand but struggled with "noisy" data or overlapping categories. Conversely, statistical regression handled noise and uncertainty effectively but produced algebraic equations that were often "psychologically meaningless" to domain experts.

The author, Carlo Berzuini, identifies two specific challenges: the need for probabilistic rather than purely deterministic expressions for real-world categories (like risk gradients), and the management of "clashes" in noisy datasets where identical examples might have different outcomes.

A Three-Phase Hybrid Pipeline

The proposed approach bridges this gap by using regression as a "numerical engine" to handle noise and then "translating" the results into logical rules. The methodology consists of three phases:

  1. The Regression Step: A standard statistical model (e.g., Logistic or Cox Regression) is trained to identify predictive relationships while filtering noise across the entire dataset.
  2. Growing a Binary Regression Tree (BRT): The algebraic constraints of the regression model are mapped into a complete binary tree. Every leaf in this tree represents a unique combination of attributes.
  3. The Simplification Step: Because a complete tree is often too complex, a simplification algorithm prunes and merges nodes to create a human-readable structure.
Figure 1: Mapping a regression equation (b) to a symbolic decision tree (c) based on predicted intervals.

A key technical contribution is the FIND-RESIDUAL algorithm. Using an object-oriented, message-passing approach, nodes communicate to identify an "optimal residual set"—a small group of outliers that do not fit clean rules. By isolating these outliers, the algorithm maintains highly consistent, simple logical rules for the majority of the population.

Clinical Case Study: Myelofibrosis (MMM)

The methodology was demonstrated using a dataset of 138 patients with Myelofibrosis with Myeloid Metaplasia (MMM), a chronic bone marrow disorder. The data included 29 clinical variables and "censored" observations (patients still alive at the study's end), which were handled using Cox’s regression model.

The process successfully distilled these variables into a simple tree with four distinct risk classes ($C_1$ through $C_4$).

Figure 5: The simplified clinical decision tree for MMM risk assessment, featuring the four risk classes and the "RESIDUAL" category.

Key Insights and Results

  • Global vs. Local Power: Unlike traditional decision tree algorithms (like ID3) that make "local" decisions at each split, this method leverages "global" statistical power—every split is informed by the entire learning sample via the initial regression.
  • The "Price" of Simplicity: The authors argue it is better to have a "Residual" category of unclassified cases than to force every patient into an inaccurate rule. In the MMM study, only 8 out of 138 patients fell into this bucket.
  • Expert Validation: Clinical experts validated the resulting tree, noting that the rules were consistent with medical intuition while providing a more precise gradation of risk than traditional diagnostic categories. For instance, patients under 45 were generally low risk ($C_1$), while older patients with severe anemia and reduced erythropoiesis fell into the highest risk category ($C_4$).

Limitations and Future Work

While efficient, the FIND-RESIDUAL algorithm provides a sub-optimal solution and does not guarantee the absolute simplest tree. Furthermore, when merging nodes from different "parents," the tree structure technically becomes a "lattice," introducing disjunctions (OR statements) that warrant further exploration. Despite these complexities, the framework remains a pioneering effort in making high-performance statistical models accessible and actionable for human experts.

Summary updated

BibTeX
@inproceedings{berzuini1988combining,
  author = {Carlo Berzuini},
  title = {{Combining Symbolic Learning Techniques and Statistical Regression Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1988},
}
Figure