This 1986 classic from the AAAI proceedings takes us back to the "Golden Age" of Expert Systems. While the specific technologies have evolved, the architectural challenges Paul J. Kline and Steven B. Dolins describe remain strikingly relevant to anyone designing complex decision-support systems today.
Introduction
In the mid-1980s, building an expert system was more of an art than a science. Knowledge engineers faced a dizzying array of choices: Should they use forward or backward chaining? How should they handle uncertainty? Is a "deep" causal model necessary, or will "shallow" heuristic associations suffice?
The authors noted that there was no "manual" for these decisions. While they had previously developed a list of 47 "Probing Questions" to help engineers interview domain experts, the list was overwhelming—engineers were "losing sight of the forest amid all of the trees." This paper seeks to synthesize those questions into a high-level framework that explains why certain AI techniques work for certain problems.
The Five Core Problem Features
The authors identified five fundamental "problem features" that dictate which AI implementation techniques are appropriate for a given task. By moving beyond a checklist of 47 specific questions to this framework, the authors provide a roadmap for matching domain problems with the right computational strategies.
1. Information Arrival Time
This feature asks: Does the information needed to solve the problem appear at program-design time, data-input time, or only during problem-solution time? For example, if you know which variables are important at design time, you can use simple forward chaining; if you don't know until the user starts typing, you might need more complex constraint propagation.
2. Evidence-Hypothesis Connection
Is the link between data and a conclusion positive (confirmation), negative (exclusion), or hierarchical? This determines whether the system needs "deep" causal reasoning or can rely on "shallow" heuristic associations.
3. Definition of a "Solution"
Does the system need to find one "good enough" solution, the single optimal solution, or all possible solutions? This dictates the stopping criteria and search strategy.
4. Environmental Accommodation
This feature considers the nature of the user and the operating environment. Is the user cooperative, knowledgeable, or potentially hostile (e.g., in military applications where an enemy might provide misleading data)? This influences the need for consistency checking and "reasoning about uncertainty."
5. Resource Wisdom
How does the system avoid a "combinatorial explosion"? This involves choosing between goal-driven (backward chaining) or data-driven (forward chaining) search to ensure computational effort isn't wasted.
Methods and Validation
The authors performed a meta-analysis of their previous work, categorizing their 47 Probing Questions—derived from literature reviews and feedback from veteran expert system builders—into these five overarching issues. They then tested this framework by seeing how well these five issues could characterize nearly 100 different AI techniques and implementation strategies.
Their "data" included landmark systems of the era, such as R1 (VAX configuration), MYCIN (medical diagnosis), and ISIS (job-shop scheduling).
Key Results
- Categorization Success: The authors found that 85% of their original 47 Probing Questions fell into one of these five categories.
- Technique Mapping: These five issues successfully characterized the appropriate use of approximately two-thirds of the nearly 100 AI techniques in their collection.
- Design Refinement: The framework allowed the authors to identify flaws in their own previous questions. For instance, they realized that a question about "guessing crucial steps" was too vague and needed to specify when that guess happens (at design time vs. solution time).
Limitations and Future Work
While the framework is robust, the authors admit that 15% of their Probing Questions did not fit into the five categories, and they could not find significant commonalities among those remaining. Furthermore, the Probing Questions were originally designed to verify a specific "target set" of AI techniques.
While the features of the problems are stable, the techniques (like the specific search algorithms of 1986) have changed. Future work would involve mapping these five features to modern paradigms like deep learning or probabilistic graphical models.
Summary updated