Designing Scripts to Guide Users in Modifying Knowledge-Based Systems

AAAI 1999

TL;DR

This paper introduces "Knowledge Acquisition (KA) Scripts" as a way to guide users through the complex process of modifying knowledge-based systems (KBSs). By analyzing how users actually change KBSs and the underlying structure of these changes, the authors developed a systematic method for creating a comprehensive library of these guiding scripts, leading to significant time savings for users.

What Problem Does This Paper Address?

Maintaining and extending knowledge-based systems (KBSs) is a crucial but challenging task. When users modify one part of a KBS, it can have unforeseen ripple effects across the entire system. It's difficult for users to anticipate and manage all these implications, potentially leading to inconsistent or broken systems. Existing tools often lack the intelligence to guide users through these complex modification sequences. This paper tackles this problem by proposing "KA Scripts" – predefined procedures that help users navigate the intricate steps involved in making changes to a KBS, ensuring that all ramifications are addressed.

Methods

The authors employed a multi-pronged approach to design and develop their KA Script library:

  • Analysis of Syntactic Changes: They began by exhaustively analyzing every possible syntactic modification to KBS elements and their potential follow-up actions. This resulted in a complete but overly general set of scripts.
  • Analysis of KBS Modification Scenarios: To gain more practical insights, they analyzed actual modification scenarios from real-world KBS development. This revealed patterns of related changes and provided a more context-specific understanding, yielding more useful scripts but not a complete set.
  • Attribute-Based Script Generation: Combining the strengths of the previous two analyses, they developed a set of attributes to describe KA Scripts. These attributes capture the "change to follow up" (element, type, transformation), "interdependency" (related KB elements), "strategy" (how to propagate the change), and "base element" (what to use as a template). This attribute-based approach allows for systematic generation and organization of a comprehensive library.

Datasets

The paper describes two case studies used for analysis:

  • Case Study I: Four successive versions of a trafficability KBS during initial prototype development.
  • Case Study II: Six successive versions of an air campaign plan evaluation KBS during extension of an implemented prototype.

For evaluation, they used:

  • Controlled Experiments: Subjects modified KBSs with and without their tool (ETM) on scenarios of varying complexity.
  • Realistic and Unseen Scenarios: Modifications from the HPKB DARPA project, specifically five extensions to a large KBS for evaluating enemy workarounds. These scenarios and the domain were unknown to the script library designers beforehand.

Key Insights and Ideas

  • Conceptual vs. Syntactic Changes: Focusing on the conceptual intent behind a change (e.g., generalizing a goal) is far more effective for guiding users than just looking at syntactic changes (e.g., changing a parameter value). Conceptual changes are closer to how users think and reason.
  • Interdependencies are Key: Understanding how different elements within a KBS are interdependent is crucial for designing effective follow-up procedures. KA Scripts should explicitly address these relationships.
  • Attribute-Driven Library Design: A systematic approach using well-defined attributes (change type, interdependency, strategy, base element) is essential for building a comprehensive and organized KA Script library. This allows for both coverage and efficient generation.
  • Scripts as Checklists: Beyond speeding up modifications, KA Scripts can act as valuable checklists, ensuring that users complete all necessary changes and leave the KBS in a coherent state.

Key Results

  • Time Savings: Controlled experiments showed a significant reduction in modification time, ranging from 15% to 52%.
  • Coverage and Usefulness: The attribute-based approach led to a library that could cover most situations in real modification tasks. In a realistic scenario, 14 out of 16 method modifications were achieved with the guidance of 12 KA Scripts.
  • Scalability: Scaling up the KA Script library did not degrade performance, indicating the robustness of the attribute-based generation method.

Limitations and Future Work

  • Completeness vs. Usefulness Trade-off: The initial analysis of syntactic changes provided completeness but lacked usefulness. The analysis of scenarios provided usefulness but not completeness. The attribute-based method aimed to bridge this gap.
  • Generalization of Operators: While general operators were identified for implementing script steps (e.g., "generalize method"), further work could explore a richer set of such operators.
  • User Familiarity: The controlled experiments showed significant time savings even with subjects familiar with the underlying system but not the KA tool. Future work could explore performance with users completely new to both.
  • Broader Application: The paper implemented scripts for goal expressions and method capabilities. Future work could extend this to other KBS elements and explore more complex interdependencies.

Summary updated

BibTeX
@inproceedings{1999designing,
  title = {{Designing Scripts to Guide Users in Modifying Knowledge-Based Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure