TL;DR
This paper introduces an explanation facility for a grammar writing system (GWS) used in machine translation. It demonstrates how to provide useful explanations (why, how, what-if) by leveraging the system's derivational history, even without supplementary knowledge, paving the way for better user understanding and acceptance of complex NLP systems.
What Problem Does This Paper Address?
Expert systems have become increasingly prevalent, and a key factor in their success is their ability to explain their reasoning to users. This allows users to understand how conclusions are reached, build trust in the system, and debug its knowledge base. However, grammar writing systems (GWS), particularly those used in Natural Language Processing (NLP) for machine translation, have traditionally lacked such explanation facilities. This is often because their users (linguists, translators, grammar writers) are expected to have a deep understanding of low-level rule traces or the systems operate in batch mode. This paper addresses this gap by proposing and implementing an explanation facility for a GWS, aiming to make these complex systems more accessible and understandable to a wider range of users, thereby improving end-user acceptance.
Methods
The core of the proposed explanation facility is based on trace-based explanation, which reconstructs the system's inferencing process. The JEMAH system, a grammar writing system associated with a machine translation environment, was used as the basis for this study. The key innovation lies in how the derivational history of the translation process is captured and utilized. Instead of just storing raw rule traces, the system records a sequence of "events," where each event represents a single transformation on the tree structure of the text being translated. Each event captures the resultant tree structure, the applied rule, the pivot node, deleted/created/modified nodes, and their annotations.
This rich derivational history is then processed to answer various types of queries:
- Why/How queries: These are answered by tracing the application of specific rules or the attachment of linguistic structures within the derivational history.
- What queries: These can be answered by examining the grammar rules and variables themselves.
- What-if (simulation) queries: The system supports this by allowing users to backtrack to a specific point in the translation process, reset values, and restart the process.
To enhance the clarity of explanations, the system also utilizes special files containing canned-text. These files provide more meaningful descriptions of grammar rules and variables, going beyond simple names to include details like rule type, keywords, and multi-level descriptions.
Datasets
Not discussed in the paper. The paper focuses on the design and implementation of an explanation facility for an existing system (JEMAH) rather than on training or evaluating on specific datasets.
Key Insights and Ideas
- Derivational History as a Rich Source: The paper highlights that the derivational history of a translation process, when structured as a sequence of transformation events, contains a wealth of information that can be effectively used for explanation, even without explicit meta-knowledge.
- Bridging the Gap for NLP Users: By providing explanation capabilities, complex GWS can become more accessible to linguists, translators, and grammar writers, fostering better understanding and interaction with the system.
- Canned-Text for Meaningful Explanations: The use of supplementary canned-text files for rules and variables significantly improves the quality and comprehensibility of explanations, making them more than just technical traces.
- Trace-Based Explanation is Sufficient (Initially): The study demonstrates that a robust trace-based explanation facility can address traditional "why," "how," and "what-if" queries without requiring complex supplementary knowledge bases, aligning with the approach of Wick and Slagle (1989).
Key Results
The paper describes the successful implementation of an explanation facility for the JEMAH GWS. This facility can answer:
- Inferencing process queries: Explaining why a rule was applied or how a specific linguistic structure was formed.
- Grammar queries: Explaining the function of rules or how they affect specific linguistic elements.
- Control strategy queries: Explaining why a particular rule was not applied.
- Simulation queries: Allowing users to explore "what-if" scenarios by modifying conditions and re-running parts of the translation process.
The system provides explanations through a simplified user interface with options to query rules, variables, source words, and target words.
Limitations and Future Work
The authors acknowledge that the current explanation facility is tightly coupled to the execution trace and the grammar knowledge base. They identify the need for supplementary knowledge to further improve the quality and range of explanations. Future work will focus on incorporating:
- Linguistic knowledge models: Similar to psychological models used in medical expert systems, this would involve knowledge about specific grammars and general linguistics.
- Parsing/generation strategy models: This meta-knowledge, potentially derived from the pre-compilation process, could help explain the system's control strategy more comprehensively.
- Hierarchical tree structures: Similar to Xplain's refinement structure, this could organize knowledge for more sophisticated explanations.
- Meta-knowledge about variables: Information on mutual exclusion, equivalent sets of values, and hierarchical relationships between variables could enhance explanations.
Summary updated