Question Answering System using Multiple Information Source and Open Type Answer Merge

Seonyeong Park, Soonchoul Kwon, Byungsoo Kim, Sangdo Han, Hyosup Shim, Gary Geunbae LeeNAACL 2015

TL;DR

This paper introduces a novel question answering (QA) system that combines the strengths of knowledge base (KB) lookups and information retrieval (IR) from raw text. By developing an "open semantic answer type detector," the system can intelligently merge answers from these disparate sources, leading to more comprehensive and accurate responses than systems relying on a single method.

What Problem Does This Paper Address?

Traditional question answering systems often fall into two camps: Knowledge Base QA (KBQA) and Information Retrieval QA (IRQA). KBQA systems, while precise, struggle with the fragmented nature of knowledge bases and lack contextual information. IRQA systems, on the other hand, can leverage context from raw text but often suffer from lower precision and difficulty in disambiguating answers. The core challenge this paper addresses is how to effectively combine these two approaches to overcome their individual limitations and provide a more robust QA system that can handle both natural language questions and keyword queries. The authors highlight that existing systems often rely on pre-determined answer types, which can limit their flexibility and accuracy.

Methods

The proposed system integrates multiple strategies and information sources. The core innovation lies in its approach to merging answers from KBQA and IRQA.

  • Multi-Strategy Approach: The system supports three types of queries: full natural language sentences, keyword sequences, and questions that can be answered by KB lookups.
  • Multi-Source Information: It leverages curated knowledge bases (like DBpedia and Freebase), raw text, auto-generated triples, and NLP results.
  • Open Semantic Answer Type Detector: This is the key novel component. Instead of relying on fixed answer types, it dynamically determines the semantic type of the answer by:
  • Feature Extraction: Identifying the main verb and focus of the question using dependency parsing and PoS tagging.
  • Mapping Property: Measuring semantic similarity between the question's verb and DBpedia properties to find the most relevant relationship.
  • Finding Semantic Answer Type: Inferring the answer type based on whether the focus is the subject or object of the identified DBpedia property. If direct inference fails, it falls back to a classifier and a transformation table.
  • Integration of KBQA and IRQA: The system first retrieves answer candidates from both KBQA and IRQA. The semantic answer type detector then compares the detected answer types of the query with the types of the candidates. Finally, it uses semantic relatedness between the answer sentence and candidates to select the best answers.
  • Lexico-semantic Pattern Matching: For IRQA, they employ patterns that match lexical, POS, and chunk-type features of a question to extract information from templates.
  • Open Information Extraction: To enrich the knowledge available, they extract triples from web text (like Wikipedia) using dependency parse trees and semantic role labeling.

Datasets

  • DBpedia: Used as a primary knowledge base.
  • Freebase: Also mentioned as a knowledge base.
  • Wikipedia Dump: Used for open information extraction.
  • Custom Dataset: The paper mentions a dataset of 670 keyword sequences and 4521 sentences used for evaluating their keyword/sentence detection.

Key Insights and Ideas

  • The Power of Merging: The fundamental insight is that combining KBQA and IRQA, rather than treating them as separate entities, leads to a more comprehensive and accurate QA system.
  • "Open" Answer Types are Crucial: Moving away from pre-defined answer types to dynamically detecting semantic answer types allows for greater flexibility and better matching of answers to diverse questions.
  • Context from Text Complements KB Precision: While KBs offer precision, the contextual information from raw text is vital for understanding nuances and providing more complete answers.
  • Leveraging Ontologies for Semantic Understanding: Using the DBpedia class type hierarchy as an open semantic answer type set provides a rich structure for understanding and matching entities.

Key Results

  • Keyword/Sentence Detection: The system achieved high accuracy in distinguishing between keyword queries and full sentences, with 96.27% for keywords and 98.12% for sentences in a five-fold cross-validation.
  • Improved Answer Merging: While specific quantitative results for the integrated QA system's overall performance are not detailed in this excerpt, the paper emphasizes that their method improves upon previous systems by using an open semantic answer type for merging.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that not all entities in KBs have specific type information, which can reduce precision if solely relied upon.
  • KBQA systems can fail due to incorrect entity or predicate disambiguation, leading to wrong answer types.
  • The IRQA system might find different sentences due to information loss during triple extraction, even if they contain the answer.
  • Future Work:
    • The authors state that this is their baseline system and they are currently exploring "textual entailment technology to improve merging accuracy."

Summary updated

BibTeX
@inproceedings{park2015question,
  author = {Seonyeong Park and Soonchoul Kwon and Byungsoo Kim and Sangdo Han and Hyosup Shim and Gary Geunbae Lee},
  title = {{Question Answering System using Multiple Information Source and Open Type Answer Merge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure