Deriving Expectations to Guide Knowledge Base Creation

AAAI 1999

TL;DR

Building large knowledge bases (KBs) is notoriously difficult due to the sheer complexity and interdependencies involved. This paper introduces a novel approach for knowledge acquisition (KA) tools that actively derives expectations to guide users during the creation of new KBs, rather than relying on pre-existing knowledge. This proactive guidance significantly speeds up the KA process and helps prevent errors early on.

What Problem Does This Paper Address?

Developing knowledge-based systems (KBS) hinges on the creation of comprehensive knowledge bases (KBs). Traditional KA tools excel when a KB already has a substantial foundation, using existing knowledge to predict what users might need to add next. However, this support breaks down when building a KB from scratch or undertaking major extensions. At these early stages, developers struggle with:

  • Designing and creating numerous KB elements: Keeping track of all definitions and anticipating future needs is overwhelming.
  • Missing pieces of knowledge: It's hard to visualize the complete KB and identify what's still needed.
  • Predicting relationships: The interdependencies between knowledge pieces are often in the developer's mind, not yet explicitly captured.
  • Inconsistencies: Detecting conflicts between newly defined elements is a painful, iterative process of running the system and debugging.

These challenges are amplified in large-scale, real-world domains, making KB development an unmanageable task. The paper aims to provide KA tools that can offer meaningful support even in these nascent stages.

Methods

The core of this paper's contribution lies in its approach to deriving expectations for KA tools, even when there's little existing knowledge. This is achieved by analyzing the KB creation task itself and identifying several sources of expectations:

  1. Enforcing constraints in the knowledge representation system: The tool checks for local errors within a newly defined KB element (e.g., undefined variables, inconsistent definitions). It also leverages existing KB elements that are similar to the one being built, using concept hierarchies to suggest related elements and guide the new definition.
  2. Looking for missing pieces of knowledge: The tool analyzes the relationships between methods based on their capabilities and subgoals. It constructs "method-submethod relation trees" to identify unmatched subgoals, prompting the user to define the missing methods. It also flags unused KB elements and suggests potential uses.
  3. Incrementally working out inter-dependencies: The system builds "capability trees" to represent hierarchies of goals and their relationships, helping users understand sub-problems. It also allows for user-defined relationships and documentation to organize methods.

The authors implemented a KA tool called EMeD (EXPECT Method Developer) within the EXPECT framework to demonstrate this approach. EMeD uses these derived expectations to guide users in creating problem-solving methods.

Datasets

Not discussed in the paper. The paper describes experiments using a "Workarounds domain" for KA tasks, but no specific datasets are detailed.

Key Insights and Ideas

  • Proactive guidance is key for KB creation: The most significant insight is that KA tools can provide substantial value before a KB is well-formed, by actively generating expectations based on the structure of the task and the nascent KB itself.
  • Leveraging "local" information: Even with limited global knowledge, analyzing local inconsistencies, concept hierarchies, and method dependencies can yield powerful expectations.
  • The "missing piece" problem is solvable: By analyzing method call graphs and identifying unmatched subgoals, KA tools can effectively highlight what needs to be built next, turning a daunting task into a manageable one.
  • Early error detection is crucial: The paper emphasizes that catching errors early, before they propagate, is far more efficient than debugging a large, complex system. EMeD's approach significantly reduces the number of errors that reach the final inference stage.

Key Results

The paper reports on preliminary evaluations comparing EMeD with a version lacking its expectation-generating features.

  • Average time savings of 30%: Subjects using EMeD were able to complete KA tasks 30% faster on average.
  • Comparable number of methods created: Users built a similar number of methods with and without EMeD, indicating that the time savings were due to increased efficiency, not reduced scope.
  • High utility of "Undefined Methods" list: This feature was the most frequently used component of EMeD, suggesting its critical role in guiding users.
  • Effective error detection: EMeD's method definition checker was used frequently and effectively detected local errors within method definitions.

The authors expect even greater time savings for domain experts with less KA experience.

Limitations and Future Work

  • User experience with complex visualizations: Subjects found the "Method Sub-method Relation Tree" and "Capability Tree" difficult to read due to the amount of information displayed. The authors plan to improve selective display and organization.
  • Limited KB size in experiments: The Search Methods and Method Organizer features were underutilized due to the small KB size in the experiments. Their utility is expected to increase with larger KBs.
  • Context-independent expectations: Currently, EMeD derives expectations regardless of the specific problem context. Future work will focus on deriving context-dependent expectations, as method relationships can change based on the problem being solved.
  • Extending to other KA tasks: The authors plan to extend EMeD to derive expectations for solving particular problems and to support collaborative KA.

Summary updated

BibTeX
@inproceedings{1999deriving,
  title = {{Deriving Expectations to Guide Knowledge Base Creation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure