Does Prior Knowledge Facilitate the Development of Knowledge-Based Systems?

AAAI 1999

TL;DR

This paper empirically investigates how much prior knowledge, specifically in the form of ontologies, helps in building knowledge-based systems. They found that while ontologies do help, the degree of assistance depends heavily on the type of ontology, how it's used, and who is using it. This work introduces metrics for quantifying this "support" and provides early empirical evidence for the benefits of knowledge sharing.

What Problem Does This Paper Address?

Building knowledge-based systems (KBS) is notoriously time-consuming and labor-intensive. A key hypothesis for accelerating this process is the reuse of existing knowledge, particularly through ontologies and domain-specific knowledge bases. However, until this study, there was a lack of systematic empirical evidence to quantify how much and in what ways this prior knowledge actually facilitates KBS development. The authors aim to fill this gap by studying knowledge reuse during the High Performance Knowledge Bases (HPKB) initiative.

Methods

The core of the study involved an empirical experiment designed to measure the "support" provided by ontologies. They defined a metric for support, s(i) = k(i) / n(i), where n(i) is the total number of terms needed to encode an item (like an axiom or a test question), and k(i) is the number of those terms already present in a prior ontology.

The experiment involved two teams (Teknowledge and SAIC) building integrated KBS to answer questions about crisis management scenarios. They used parameterized question grammars to generate multiple batches of questions (SQ, TQA, TQB, TQC, TQD) over several phases of the project. The key innovation was to track the terms used by each team to encode both the test questions themselves and the axioms needed to answer them, and then compare these terms against available prior ontologies (specifically, the general Upper Ontology (UO) and the more comprehensive Cyc ontology).

Datasets

The paper does not explicitly list "datasets" in the traditional machine learning sense. Instead, the "data" for evaluation was generated through:

  • Crisis Scenario: A defined domain for the knowledge-based systems.
  • Parameterized Question (PQ) Grammar: A generative grammar used to create multiple batches of questions (SQ, TQA, TQB, TQC, TQD). This allowed for systematic testing and re-testing, and also enabled the developers to limit their system's scope to relevant topics.
  • Prior Ontologies:
    • Upper Ontology (UO): A general ontology containing broad class names, relationships, etc.
  • Cyc Ontology: A much larger and more comprehensive ontology.

The "datasets" in this context are the sets of test questions and the knowledge (terms and axioms) encoded by the system developers to address them.

Key Insights and Ideas

  • Ontologies Help, But It's Nuanced: The study confirms that prior knowledge in the form of ontologies does help in building KBS. However, the degree of help is not uniform.
  • Type of Ontology Matters: More comprehensive ontologies (like Cyc) provided more support than more general ones (like the UO). This suggests that the richness and specificity of the ontology are crucial.
  • User Familiarity and Preference: The Cyc/Tek team, who had access to and were likely more familiar with the Cyc ontology, showed higher reuse rates from it compared to the SRI team's use of the UO. The SRI team sometimes preferred defining their own terms over reusing existing ones, indicating that user preference and familiarity play a significant role.
  • Different Uses, Different Support: Prior knowledge seemed to provide less support for encoding axioms (rules for problem-solving) than for encoding the test questions themselves. The authors speculate this might be because test questions use more general terms, while axioms might involve more specific or individual-level terms less likely to be in a general ontology.
  • Reuse Rate of Domain-Specific Knowledge: For knowledge that wasn't from the general UO, the reuse rate was consistently around 60%, regardless of who developed the ontology. This suggests a stable rate of reuse for domain-specific or previously defined terms.

Key Results

  • Quantitative Support: Table 1 and Figure 3 show the "support" (reuse rate) provided by ontologies. For example, for encoding sample questions (SQ), the UO provided 22% support for SRI and 42% for Cyc/Tek. For test questions (TQA/TQC), the UO support was lower (around 15-19% for SRI), but other prior knowledge (like terms defined for SQ) provided significant support (around 59-62% for SRI).
  • Cyc/Tek vs. SRI: Cyc/Tek generally showed higher reuse rates, especially from the comprehensive Cyc ontology, compared to SRI's reuse from the UO.
  • Terms per Question: Cyc/Tek required slightly more terms per question (2.86) than SRI (2.62), but got more support from prior ontologies.
  • Axiom Encoding Support: Table 3 shows that for SRI, the support for encoding axioms from the UO was consistently low (around 0.1 to 0.11), but support from ontologies developed in earlier phases of the project (SQ, TQA) was much higher (around 0.5).

Limitations and Future Work

  • Limitations:
    • The study primarily focused on the reuse of terms, not necessarily larger knowledge fragments or complex reasoning structures.
  • The data are "crude summaries" and don't fully capture the cognitive effort or specific processes knowledge engineers undertake.
  • The findings are specific to the crisis management domain and the two research groups involved. Generalizability to other domains and groups is an open question.
  • It's unclear why prior ontologies provided less support for encoding axioms than test questions.
  • Future Work:
    • Investigate how knowledge engineers search for relevant terms and how quickly KBS diverge from ontologies.
  • Explore how larger fragments of ontologies are reused.
  • Analyze how general ontologies inform KBS design and what factors influence their adoption.
  • Conduct further studies to generalize findings to other domains and research groups.
  • The authors plan to analyze more data from the HPKB program and design new experiments for the second year to address these questions.

Summary updated

BibTeX
@inproceedings{1999does,
  title = {{Does Prior Knowledge Facilitate the Development of Knowledge-Based Systems?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}

Loading related content...

Figure