A Script-Based Approach to Modifying Knowledge Bases

AAAI 1997

TL;DR

Modifying complex knowledge-based systems (KBS) is notoriously difficult due to intricate interdependencies. This paper introduces "KA Scripts," a novel approach that captures common modification sequences to guide users, significantly reducing errors and improving efficiency in maintaining KBS.

What Problem Does This Paper Address?

Building and maintaining knowledge-based systems (KBS) has been a persistent challenge for decades. While initial prototypes are developed, subsequent modifications to adapt to changing real-world conditions or new user requirements are often complex and error-prone. These modifications can involve numerous individual changes across different components of the KBS, requiring careful coordination to prevent the system from becoming unusable. Existing knowledge acquisition tools often fall short by either limiting users to predefined roles or by only detecting inconsistencies without providing actionable guidance for resolution. The core problem is the difficulty users face in coordinating multiple, interdependent changes and ensuring the overall coherence of the KBS after modifications.

Methods

The paper proposes a "script-based approach" to knowledge acquisition, specifically for modifying KBS. The central idea is the introduction of KA Scripts (Knowledge Acquisition Scripts). These scripts are defined as prototypical sequences of changes, coupled with conditions that determine their relevance based on the current state of the knowledge base and prior modifications.

Key aspects of their approach include:

  • KA Scripts: These scripts encapsulate common modification patterns and their associated effects. They aim to guide users through a series of coordinated changes rather than treating each modification in isolation.
  • Contextual Guidance: The system leverages information about the current knowledge base, detected errors, and a history of user changes to suggest relevant KA Scripts. This context is crucial for providing more specific and helpful guidance than generic error messages.
  • Error Detection and Resolution: The system identifies problems (errors or knowledge gaps) in the KBS and then suggests KA Scripts that can resolve these issues.
  • ETM (EXPECT's Transaction Manager): This is the implementation of their script-based approach, integrated with the EXPECT architecture. ETM guides users through applying KA Scripts, offering choices and assistance in making the necessary modifications.
  • Collaborative Framework: ETM identifies potential KA Scripts, but the user ultimately decides which script is most appropriate for their intended modification.

Datasets

Not discussed in the paper. The paper uses a simplified transportation domain for examples and evaluations, but does not refer to specific, named datasets.

Key Insights and Ideas

  • Modifications are sequences, not isolated events: The paper highlights that modifying a KBS is rarely a single change but a series of interdependent steps. Treating them as such, through scripts, is key to success.
  • Context is king for effective guidance: Generic advice is less helpful than context-aware suggestions. Understanding the history of changes and the specific problem-solving context allows for more targeted and useful guidance.
  • KA Scripts as "transaction managers": The analogy to database transactions is powerful. KA Scripts help ensure that a modification "transaction" is completed coherently, preventing the KBS from being left in an inconsistent state.
  • User agency is preserved: While scripts provide guidance, the user retains control over which script to apply, acknowledging that users possess knowledge not available to the tool.
  • The difficulty of "starting" modifications: A significant insight from their evaluations was that users often need help figuring out where to begin a modification, suggesting a need for "initiation scripts" in addition to those for completing changes.

Key Results

Preliminary evaluations comparing subjects using EXPECT (without ETM) versus ETM showed promising results:

  • Reduced Modification Time: Subjects using ETM consistently took less time to complete modifications, especially in more complex scenarios. For the PAE scenario, ETM users took significantly less time (e.g., 41 minutes vs. 53 minutes for total time, and 20 minutes vs. 32 minutes for transaction completion time).
  • Increased Automation: KA Scripts automatically handled a portion of the required changes, freeing up users to focus on the core task. For example, in the RTT scenario, ETM users had 2 changes made automatically, while EXPECT users had none.
  • Fewer Errors and Overlooked Steps: Subjects using ETM were less likely to overlook parts of the modification instructions, likely because ETM's step-by-step guidance helped them keep track of pending changes.
  • User Satisfaction: Users found the guidance provided by KA Scripts understandable and helpful, with none opting to abandon ETM for the standard EXPECT tool.

Limitations and Future Work

  • Limitations:
    • The current library of KA Scripts, while comprehensive for their tested scenarios, is not exhaustive for all possible modification strategies.
  • The effectiveness of ETM might be more pronounced with users less familiar with the underlying EXPECT architecture.
  • The user interface could be improved with better visualizations and abstractions of the knowledge base.
  • Future Work:
    • Developing "initiation scripts" to help users decide where to start a modification.
  • Extending the library of KA Scripts to cover more modification scenarios.
  • Enhancing the user interface for better visualization and on-line help.
  • Further evaluation with domain experts who have no prior exposure to EXPECT or ETM.

Summary updated

BibTeX
@inproceedings{1997script,
  title = {{A Script-Based Approach to Modifying Knowledge Bases}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1997},
}
Figure