Introduction
In the field of machine learning, particularly during the early 1980s, the dominant paradigm was inductive generalization—identifying patterns across large datasets. However, this approach often faltered when faced with prediction failures. When a predicted outcome fails to materialize, such as a favored horse losing a race, the number of potential explanatory features can lead to a "combinatorial explosion." This paper introduces HANDICAPPER, a system designed to emulate human-like causal reasoning to prune irrelevant hypotheses and focus on meaningful explanations for failure.
The Combinatorial Explosion Problem
The primary challenge in learning from failure is the sheer volume of data. If a horse possesses ten unique features, a system attempting to explain a loss might generate over 1,000 combinations of those features to find a cause. Humans avoid this by applying domain-specific causal knowledge. Salzberg’s research aimed to formalize this process, using expert knowledge to restrict the search space for hypotheses from thousands of possibilities to a manageable few.
The HANDICAPPER Methodology
HANDICAPPER predicts the winners of thoroughbred horse races using a structured three-step process triggered by prediction failures:
- Feature Comparison: Upon a failure, the system identifies all features that differ between the predicted winner and the actual winner.
- Causal Filtering: Instead of exhaustive testing, the system consults a library of expert-provided "causal rules." It specifically searches for causal violations—instances where the observed data contradicts expected performance patterns.
- Hypothesis Generation: A detected causal violation becomes the primary hypothesis. This hypothesis is then indexed in memory, allowing the system to recognize and avoid similar patterns in future predictions.
Case Study: The "Too Good to Be True" Rule
A pivotal example in the development of HANDICAPPER involved the horse Well I’ll Swan. While the horse's statistics appeared excellent, it was "dropping down" in claiming price—being entered into a significantly cheaper race.
As shown in the feature comparison above, Well I'll Swan had finished well in its last two races (1st and 3rd) but was dropping down $15,000 from its last race. While a lower price usually suggests an easier win, the causal rule identifies a violation: if a horse is winning but the owner is suddenly selling it for much less, it is likely the horse is injured or ill. By identifying this "too good to be true" scenario, HANDICAPPER generates a specific hypothesis for the failure.
Results and Efficiency
The integration of causal knowledge significantly improved the system's efficiency. The reduction in the hypothesis search space was dramatic:
- Baseline: Without causal knowledge, the program generated approximately 5,000 hypotheses for the first three races.
- Basic Labeling: Assigning simple "good/bad" labels to features reduced the space to fewer than 100 hypotheses.
- Causal Reasoning: Implementing complex causal rules (such as the "dropping down" logic) reduced the output to just 8 hypotheses per race.
Furthermore, the system demonstrated successful learning. After analyzing the Well I’ll Swan failure, HANDICAPPER correctly predicted the loss of a similar horse in a subsequent race, despite its strong surface-level statistics.
Limitations and Future Work
While HANDICAPPER serves as a successful proof of concept, it faces several limitations. The study was conducted on a small sample of approximately 12 races, and the system relies heavily on a "knowledge engineering bottleneck"—it requires pre-defined expert rules and cannot yet discover new causal relationships autonomously. Future research is needed to explore how these causal structures might be learned automatically and applied to other domains like stock market analysis.
Summary updated