An Achievement Test for Knowledge-Based Systems: QUEM

IJCAI 1997

TL;DR

This paper introduces QUEM, a novel method for evaluating knowledge-based systems (KBSs) by measuring their "experience level" based on the quality of their solutions. QUEM bridges the gap between simply knowing if a system works and understanding how well it performs, offering a quantitative way to assess a KBS's competence relative to human experts.

What Problem Does This Paper Address?

Evaluating the performance of knowledge-based systems (KBSs) is a persistent challenge. While traditional methods focus on validation, verification, and testing (ensuring correctness and consistency), these don't fully capture a system's overall competence or how well it performs its intended task. It's difficult to quantify a KBS's "skill level" in a way that allows for meaningful comparisons, either between different KBSs or between a KBS and its human users. This lack of a robust, quantitative measure makes it hard for developers to gauge the effectiveness of their approaches, decide on further development, or determine how a KBS should be integrated into human workflows. The paper aims to address this by providing a method to estimate a KBS's performance on an "absolute scale" defined by human experience.

Methods

The core of QUEM is to leverage human experts as a benchmark. The process involves:

  1. Problem Solving: A set of problems is solved by both human subjects with varying levels of experience and the KBS(s) being evaluated.
  2. Solution Ranking: Expert judges (with experience exceeding that of the subjects) independently rank the quality of all solutions (human and KBS) for each problem from worst to best.
  3. Rank Adjustment: If judges assign tied ranks, these are normalized to allow for consistent comparison.
  4. Subject Averaging: The average quality rank for each human subject and each KBS is computed across all problems.
  5. Skill Function Fitting: The average quality rankings of human subjects are plotted against their years of experience. A "skill function" (typically a linear regression line) is fitted to this data, establishing a relationship between human experience and solution quality.
  6. Experience Estimation: The KBS's average quality rank is then plugged into the fitted skill function to estimate the KBS's equivalent years of human experience.
  7. Confidence Bands: Confidence bands are constructed around the skill function to provide an experience interval, indicating the range within which the KBS's true experience level likely lies with a certain confidence (e.g., 95%).

This method allows for a quantitative assessment of a KBS's performance, framing it in terms of human expertise.

Datasets

The paper describes an example evaluation using a manufacturing KBS called MACHINIST (later P³). For this evaluation:

  • Problems: Three problems of approximately the same difficulty were prepared.
  • Human Subjects: Seven subjects with experience ranging from 2 to 10 years.
  • Expert Judges: Two judges with 15 and 18 years of experience.
  • KBS: The MACHINIST (P³) manufacturing KBS.

The paper notes that for complex domains like manufacturing and software development, even simple problems can be complex enough to reveal significant differences in skill. No new benchmark datasets were introduced; existing problems were used.

Key Insights and Ideas

  • Human Expertise as a Benchmark: The most significant insight is the idea of using human experts not just as a comparison point, but as the very scale for measuring KBS competence. This elegantly sidesteps the difficulty of creating objective, quantifiable quality metrics for complex domains.
  • Experience as the Metric: Framing KBS performance in terms of "years of experience" provides an intuitive and relatable measure of sophistication, directly addressing the "how expert is my system?" question.
  • Graceful Degradation: The method is robust to variations in judge assessments or subject motivation. While noise might broaden the experience interval (reducing precision), it won't drastically alter the overall estimate.
  • Beyond Correctness: QUEM moves beyond traditional VVT metrics to assess the quality and sophistication of solutions, which is crucial for understanding a KBS's practical utility.

Key Results

In the example evaluation of the MACHINIST (P³) KBS:

  • The human subjects' average quality rankings ranged from 2.50 (2 years experience) to 7.33 (10 years experience).
  • The KBS achieved an average quality ranking of 5.67.
  • A linear skill function was fitted: y = -1.97 + 1.67x, where y is years of experience and x is average solution quality rank.
  • Plugging the KBS's rank (5.67) into the skill function yielded an estimated experience level of 7.20 years.
  • The 95% confidence interval for this estimate was 6.03 to 8.36 years of experience.

This suggests that the MACHINIST KBS, at the time of evaluation, demonstrated a level of problem-solving sophistication equivalent to a human expert with over 7 years of experience in that domain.

Limitations and Future Work

  • Domain Dependency: QUEM is most effective in domains where increased experience demonstrably leads to measurably increased solution quality. It's harder to prove the absence of such a relationship.
  • Subject/Judge Selection: Careful selection of judges and subjects with experience matching the KBS domain is critical. The range of subject experience needs to be appropriate to capture the KBS's estimated level. Pilot studies may be necessary to determine this range.
  • Problem Complexity: While simple problems can suffice, the complexity must be sufficient to reveal skill differences.
  • Future Work: The paper suggests QUEM can be used to:
  • Estimate the experience level of a single KBS.
  • Identify changes in experience level between different versions of a KBS.
  • Compare multiple KBSs within the same or different domains.
  • Estimate the impact of a computer assistant on a user's skill level.

Summary updated

BibTeX
@inproceedings{1997achievement,
  title = {{An Achievement Test for Knowledge-Based Systems: QUEM}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}
Figure