A Question Answering System Supported by Information Extraction

Rohini Srihari, Wei LiNAACL 2000

TL;DR

This paper introduces TextractQA, a question answering (QA) system that leverages a multi-level information extraction (IE) approach. By integrating Named Entity (NE) recognition, Correlated Entity (CE) extraction, and General Event (GE) extraction, TextractQA aims to move beyond simple document retrieval to provide direct answers, significantly advancing the state-of-the-art in information retrieval.

What Problem Does This Paper Address?

The paper addresses the growing need for effective question answering (QA) systems in the face of the internet's information explosion. Traditional Information Retrieval (IR) systems excel at finding documents but leave the burden of extracting specific answers to the user. Existing QA systems are often limited to toy examples or restricted domains. The authors highlight the TREC-8 QA track's goal: to stimulate research into real-world QA applications that provide direct answers, not just document pointers. They argue that Information Extraction (IE) is a crucial component for achieving this, and their work focuses on how different levels of IE can support and enhance QA capabilities.

Methods

The core of the proposed system is the Textract IE system, which employs a hierarchical, three-level architecture:

  • Level 1: Named Entity (NE) Tagging: This is the foundational level, identifying entities like persons, organizations, locations, dates, etc. Textract's NE tagger is described as state-of-the-art, achieving comparable speed and accuracy to existing systems. Notably, Textract expands the types of NEs beyond standard MUC definitions to include more granular information relevant to QA, such as durations, frequencies, and various measurements.
  • Level 2: Correlated Entity (CE) Extraction: This level focuses on extracting pre-defined relationships between identified entities. For example, for a person entity, CE can extract age, gender, affiliation, position, etc., essentially building a mini-profile. This enriches the information beyond simple entity identification.
  • Level 3: General Event (GE) Extraction: This is the highest level, aiming to extract open-ended events, including who did what (to whom), when, and where. This involves identifying verb-centered argument structures and associated temporal and locational information.

The QA system, TextractQA, utilizes these IE levels. It comprises a Question Processor (determining the asking point and expanding the question) and a Text Processor (using a search engine to narrow down documents, followed by IE processing). A Text Matcher then links the question template with the processed documents to find answers, using keyword matching, NE types, and sentence-level commonality. The question processing employs a robust strategy using both shallow parsing-based rules and simple keyword-based rules.

Datasets

The paper mentions the TREC-8 QA track as the context for their evaluation and a benchmark for their system. While specific training datasets for the IE components are not explicitly detailed, the work is heavily influenced by the MUC (Message Understanding Conferences) specifications, which have become de facto standards for IE tasks. The authors also mention the use of manually maintained word lists for org_w and person_w and an ongoing effort to automate their learning using machine learning techniques.

Key Insights and Ideas

  • IE as a Natural Partner for QA: The authors strongly advocate for the synergistic relationship between IE and IR, viewing QA as a primary intelligence that IE can offer to IR.
  • Hierarchical IE for Deeper Understanding: The multi-level IE approach (NE, CE, GE) is crucial. While NEs are helpful for many "who, what, when, where" questions, CE and GE extraction are necessary for more complex queries, such as relationship-based questions or questions about events.
  • Expanded NE Definitions are Key: Including more granular NE types (like duration, frequency) directly addresses the "how" questions that are common in QA, making the system more versatile.
  • Moving Beyond Document Retrieval: The paper emphasizes the shift from simply finding documents to extracting direct answers, a significant advancement for user experience and information access.
  • Robust Question Processing: The combination of shallow parsing rules and keyword-based rules provides a robust mechanism for understanding diverse question formulations.

Key Results

The paper reports that in the TREC-8 QA track competition, Cymfony QA achieved an accuracy of 66.0%. The authors consider this a very encouraging result, especially given that they only used NE technology to support QA in that particular run. This suggests significant potential for improvement with the integration of CE and GE levels.

Limitations and Future Work

The authors acknowledge several limitations:

  • Question Types: Currently, the system primarily handles "wh-questions." "Why" and "how" questions are more challenging as their asking points don't always map directly to NE types. They plan to extend support to "yes-no" questions by introducing CE and GE templates.
  • Data Dependency: The system's ability to answer is contingent on the information being explicitly present in the processed documents. For real-world QA portals, they suggest complementing document-based IE with other knowledge sources (e.g., yellow pages, databases).
  • Linguistic Processing Gaps: The current system lacks full sentence-level parsing and cross-sentential co-reference resolution (CO). These are planned for future development as high-level IE technology is integrated.

Future work will focus on:

  • Exploiting CE and GE: Developing a QA architecture that fully leverages CE and GE templates to answer a wider range of questions, particularly relationship-based and event-based queries.
  • Handling Semantic Equivalence: Researching how to match questions with different verbs but similar meanings to extracted events.
  • Addressing Out-of-Scope Questions: Investigating how to handle questions that ask about information not covered by GE or CE.

Summary updated

BibTeX
@inproceedings{srihari2000question,
  author = {Rohini Srihari and Wei Li},
  title = {{A Question Answering System Supported by Information Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2000},
}
Figure