Knowledge-Based Question Answering

Chin-Yew Lin, Fabio Rinaldi, James Dowdall, Michael Hess, Diego Mollá Aliod, Rolf Schwitter, Kaarel KaljurandKes 2003 · DOI

TL;DR

This paper introduces ExtrAns, a system for knowledge-based question answering in technical domains. It tackles the challenge of accessing information within large technical documents by building a domain-specific knowledge base from the text itself, focusing on understanding the nuances of technical terminology.

What Problem Does This Paper Address?

Accessing specific information within vast amounts of technical documentation is a significant challenge. While general question-answering (QA) systems exist, they often struggle with the specialized vocabulary and complex relationships found in technical fields. Traditional QA systems, like those evaluated in TREC, often rely on shallow text analysis due to the sheer volume of data. This paper aims to bridge this gap by developing a system that can effectively understand and query technical documents by first building a rich, domain-specific knowledge base. The core problem is how to represent and leverage technical terminology to enable accurate and efficient information retrieval in response to natural language questions.

Methods

The ExtrAns system employs a two-phase approach:

  • Offline Phase (Knowledge Base Creation):
    • Terminology Extraction: Technical terms are identified from the source documents.
  • Term Processing: A sophisticated approach is used to handle multi-word technical terms. This involves packing multi-word units into single lexical tokens before syntactic analysis, which significantly reduces parsing complexity (almost 50% reduction reported).
  • Terminological Knowledge Base (TermKB) Construction: A computational thesaurus is built, organizing terms around synonymy and hyponymy. This includes:
  • Strict Synonymy: Variations in punctuation, acronyms, or orthography.
  • Lexical Hyponymy: Identified by pattern matching, where a term is a more specific version of another (e.g., "cargo compartment door" is a lexical hyponym of "door").
  • WordNet Hyponymy: Leveraging WordNet's hierarchical structure to map semantic relationships onto the domain terminology.
  • Semantic Representation: Documents are parsed using a Link Grammar parser, and dependency-based syntactic structures are transformed into Minimal Logical Forms (MLFs). These MLFs encode semantic dependencies between open-class words and prepositional phrases.
  • Online Phase (Question Answering):
    • Query Analysis: User queries are also parsed into MLFs.
  • Knowledge Base Matching: The query MLF is matched against the stored MLFs in the document knowledge base.
  • Answer Extraction: Sentences whose MLFs best match the query MLF are identified.
  • Terminological Normalization: Synset identifiers are used to replace terms, allowing queries to automatically map to all their variants (synonyms, related terms).
  • Relaxing the Query: If an exact answer isn't found, the system can "relax" the query by including hyponyms and hyperonyms of terms. If still no exact match, it returns the sentence with the semantically closest MLF to the query MLF.

Datasets

  • Aircraft Maintenance Manual (AMM) of the Airbus A320: This was a primary dataset used for developing and testing the system. Its size (120MB) is noted as being smaller than TREC-QA corpora, allowing for more sophisticated NLP techniques.
  • Linux HOWTOs: This is mentioned as a new target domain for future experiments, chosen for its open-source nature, allowing for wider availability of results.
  • WordNet: Used as a source for semantic relations to augment the domain terminology.

Key Insights and Ideas

  • The Critical Role of Terminology: The paper strongly emphasizes that technical domains are heavily reliant on precise terminology. Effectively handling multi-word technical terms is crucial for computational accuracy and efficiency.
  • Pre-parsing Terminology Normalization: A key innovation is packing multi-word terms into single lexical tokens before syntactic analysis. This significantly simplifies parsing and improves efficiency, a departure from typical approaches that might handle terms post-parsing.
  • Leveraging Domain-Specific Knowledge: Unlike general QA systems, ExtrAns capitalizes on the structured nature of technical documents and the specific domain terminology to build a more accurate and relevant knowledge base.
  • Flexible Answer Retrieval: The system's ability to "relax" queries by incorporating ontological information (hyponymy/hyperonymy) and to find semantically closest answers when exact matches fail is a practical approach for dealing with the inherent ambiguities and variations in natural language.

Key Results

  • Reduced Parsing Complexity: Packing multi-word units into single lexical tokens reduced parsing complexity by almost 50%.
  • TermKB Validation: Manual validation of 500 synsets found 1.2% inappropriate terms, and validation of 500 lexical hyponymy links found them all valid. However, WordNet hyponymy links showed a higher invalidity rate (over 35%). This led to excluding WordNet hyponymy from the final TermKB.
  • Qualitative Demonstration: The paper provides an example interaction (Figure 4) showing how a query about the "distribution network" can be answered by retrieving relevant sentences from the AMM, highlighting the specific terms used for deduction.

Limitations and Future Work

  • WordNet Hyponymy Issues: The paper notes that WordNet hyponymy links were found to be largely invalid when applied to the technical domain terminology, suggesting that direct translation of general semantic relations might not always be accurate.
  • Inference Limitations: The current inference capabilities are limited to a "small experiment." Large-scale extension would require addressing domain-specific inferences, contradictory knowledge, inference cycles, and the broader problem of knowledge acquisition, potentially necessitating a domain ontology or more general world knowledge.
  • Handling Novel Term Variants: While the system can handle variations seen offline, it acknowledges that users might introduce entirely new term variants. The proposed solution of filtering queries using FASTR is a step towards addressing this.
  • Future Work: Exploring the use of ontological information to compute semantic closeness based on ontological distance between words is suggested as a future avenue.

Summary updated

BibTeX
@inproceedings{lin2003knowledge,
  author = {Chin-Yew Lin and Fabio Rinaldi and James Dowdall and Michael Hess and Diego Mollá Aliod and Rolf Schwitter and Kaarel Kaljurand},
  title = {{Knowledge-Based Question Answering}},
  booktitle = {Kes},
  year = {2003},
  doi = {10.1007/978-3-540-45224-9\_106},
}
Figure