Sure, here's a blog-style summary of the paper "Comparing Expert Systems Built Using Different Uncertain Inference Systems":
TL;DR
This study compares the usability and accuracy of six different methods for handling uncertainty in expert systems. It found that rule-based systems like EMYCIN and PROSPECTOR were less accurate when dealing with conflicting evidence, suggesting that their knowledge representation methods may be more challenging for users to tune effectively.
What Problem Does This Paper Address?
Expert systems are designed to mimic human decision-making, often in domains where information is incomplete or uncertain. To handle this uncertainty, various "Uncertain Inference Systems" (UISs) have been developed, each with its own approach to representing and manipulating uncertain knowledge. However, for a long time, there was little empirical evidence to guide the selection of the "best" UIS for a particular application. While theoretical arguments abound, the practical usability and real-world accuracy of these systems remained largely untested. This paper aims to fill that gap by empirically comparing the intuitiveness and accuracy of several prominent UISs.
Methods
The researchers designed a controlled experiment where 60 participants, with varying levels of AI experience, were trained on a hypothetical diagnostic problem. After reaching a criterion level of performance, they were randomly assigned to one of six UISs: EMYCIN, PROSPECTOR, Dempster-Shafer theory, Fuzzy set theory, simplified probability theory (assuming marginal independence), and linear regression using probability estimates. Participants then built, tested, and refined an expert system using their assigned UIS. The study focused on measuring both the "usability" (how easily participants could tune their systems) and the "accuracy" of the resulting systems.
Datasets
The study did not use pre-existing datasets. Instead, a hypothetical diagnostic problem was created. The problem involved diagnosing a malfunctioning machine based on temperature and pressure readings. The data for learning and testing trials were generated according to a predefined contingency table that specified the probabilities of different evidence combinations and their relation to the machine's state (working or malfunctioning). This allowed for controlled generation of "consistent" (both readings normal or both high) and "mixed" (one normal, one high) evidence scenarios.
Key Insights and Ideas
- Usability vs. Accuracy Trade-off: The study suggests that while some UISs might be easier to tune (like linear regression), this doesn't automatically translate to higher accuracy, especially in complex scenarios.
- Rule-Based Systems Struggle with Mixed Evidence: EMYCIN and PROSPECTOR, which rely on rule-based knowledge representation, showed significantly lower accuracy when presented with conflicting evidence (mixed temperature/pressure readings).
- Parameter Estimation Challenges: The difficulty in tuning EMYCIN and PROSPECTOR systems for mixed evidence cases appears to stem from the challenge of accurately estimating parameters within their rule-based structure, rather than simply overlooking these cases.
- Structured Approaches May Be More Robust: More structured methods, like simplified probability and linear regression, seemed to handle mixed evidence more effectively, possibly because their parameter estimation is more straightforward.
Key Results
- Tuning Effort: Participants using linear regression required the fewest tuning trials, while those using PROSPECTOR needed the most. However, this difference was largely explained by the number of parameters each UIS required.
- Overall Accuracy: All UISs could be used to build reasonably accurate systems for the simple problem domain.
- Mixed Evidence Performance: Systems built with EMYCIN and PROSPECTOR were significantly less accurate when diagnosing problems with mixed evidence (one normal, one high reading) compared to consistent evidence. For mixed evidence, their proportion of correct diagnoses was around 0.26-0.31, while other UISs achieved 0.53-0.74.
- Parameter Overestimation: EMYCIN and PROSPECTOR systems tended to overestimate the likelihood of malfunction in mixed cases, with average parameter estimates for these scenarios being higher than those from other UISs.
Limitations and Future Work
The authors acknowledge several limitations:
- Simplicity of the Problem Domain: The hypothetical problem only involved two pieces of evidence. In more complex real-world scenarios with numerous interacting factors, humans might find it easier to encode simple rules, potentially diminishing the disadvantage of rule-based UISs.
- Expertise vs. Usability: The study focused on the "inherent usability" of UISs for individuals learning a new domain. It doesn't necessarily reflect the optimal performance achievable by experienced system developers.
- Nature of Knowledge: The knowledge captured was for a simple diagnostic task. Further research is needed to understand how different UISs handle the encoding of more complex, practical expertise.
Future work could explore these limitations by investigating more complex problem domains and by comparing the performance of experienced knowledge engineers using these UISs.
Summary updated