Acquiring Problem-Solving Knowledge from End Users: Putting Interdependency Models to the Test

AAAI 2000

TL;DR

This paper introduces and evaluates EMeD, a knowledge acquisition (KA) tool that uses "Interdependency Models" to guide non-expert users in building AI systems. Experiments with domain experts show that EMeD significantly speeds up knowledge entry and allows users to complete more tasks compared to a version without these guiding models, suggesting a promising path for making AI development more accessible.

What Problem Does This Paper Address?

Building AI systems often requires encoding complex "problem-solving knowledge" – essentially, how to achieve a goal. Traditionally, this has been the domain of highly trained knowledge engineers. However, there's a growing need for tools that allow "end users" (people without formal AI or computer science backgrounds) to contribute this knowledge directly. Existing tools often fall short, either by being too complex, only handling basic knowledge types (like defining objects), or requiring users to provide numerous examples, which isn't always feasible. This paper tackles the challenge of creating KA tools that are usable and effective for these end users, specifically focusing on how to guide them in specifying problem-solving knowledge.

Methods

The core innovation presented is the use of Interdependency Models (IMs) within the EMeD (EXPECT Method Developer) KA tool. IMs represent relationships between different components of a knowledge base (e.g., how defining a constraint might necessitate defining how to fix it). EMeD leverages these IMs in several ways:

  • Method Proposer: Suggests missing knowledge components and provides initial sketches for new methods.
  • Method Sub-method Analyzer: Identifies missing links and suggests how methods can be combined.
  • Method-Capability Hierarchy: Helps users find existing methods that can achieve a desired capability.
  • Global-Error Detector: Analyzes the knowledge base for inconsistencies.

The paper describes a controlled experimental design to evaluate EMeD's effectiveness. This involved comparing EMeD against an "ablated version" (identical interface but without IM-based guidance) in a within-subject design. Participants performed knowledge base modification tasks. A preliminary study with a diverse range of users (from CS experts to those with no technical background) helped refine the tool and experimental methodology.

Datasets

Not discussed in the paper. The paper describes using a knowledge-based system for critiquing military courses of action as the domain for the experiments. Participants extended this existing knowledge base.

Key Insights and Ideas

  • Interdependency Models are Key for End Users: The most significant finding is that IMs provide crucial guidance for non-expert users, enabling them to understand how new knowledge fits into the existing system and to complete more complex tasks.
  • Bridging the Gap to End Users: The research demonstrates that with appropriate tools and guidance, end users can contribute to building problem-solving knowledge, not just static domain knowledge.
  • Training Time is Manageable: Even with a significant amount of training (8 hours for the preliminary study), end users were able to learn the system and perform tasks. The authors suggest this could be further reduced with improved interfaces and online help.
  • Structured Editors are Essential: For users with less technical background, a structured editor that guides syntax through point-and-click operations significantly reduces errors and makes the process more accessible.
  • Visualizing Relationships is Crucial: Users found it easier to see interrelationships with EMeD, particularly through features like the Method Sub-method Analyzer.

Key Results

  • Faster Knowledge Entry: While not explicitly quantified in the main experiment, the preliminary study showed that users with less technical background benefited more from EMeD, and the tool saved time.
  • Increased Task Completion: In the experiment with domain experts, users accomplished almost twice as many tasks with EMeD (utilizing IMs) compared to the ablated version.
  • More Complex Tasks Achieved: None of the subjects could complete the more complex modification tasks using the ablated version, whereas EMeD enabled completion.
  • High Usefulness Ratings: Features like the Method Proposer (suggesting missing knowledge and sketches) and the Method Sub-method Analyzer (visualizing relationships) received high usefulness ratings (4.7/5 and 4.3/5 respectively).

Limitations and Future Work

  • Small Sample Size: The main experiment with domain experts had only four participants, limiting the statistical significance of the results, although initial experiments showed statistical significance.
  • Training Time: While manageable, the 8 hours of training in the preliminary study could be a barrier. Future work could focus on reducing this through interface improvements and better online help.
  • User Differences: The authors acknowledge that users with different backgrounds perform differently, and more research is needed to understand these variations and tailor tools accordingly.
  • Goal Composition Challenges: Expressing complex relationships and composite goals remained challenging for some users, suggesting a need for more advanced guidance in these areas.
  • Proactive Guidance: While EMeD is proactive, future tools could benefit from even better visual cues and pop-up windows to guide users, especially those further removed from the technical details. Integrating concepts like "Knowledge Acquisition Scripts" could also help users track their progress on complex changes.

Summary updated

BibTeX
@inproceedings{2000acquiring,
  title = {{Acquiring Problem-Solving Knowledge from End Users: Putting Interdependency Models to the Test}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2000},
}
Figure