Answer Extraction

Michael Collins, Steven P. Abney, Michael Collins 0001, Amit SinghalAnlp 2000 · DOI

TL;DR

This paper introduces a question-answering system that goes beyond retrieving entire documents to extract precise, short answers. By combining robust passage retrieval with sophisticated linguistic processing for entity extraction and classification, the system significantly improves answer precision, especially for short answer requirements.

What Problem Does This Paper Address?

Traditional Information Retrieval (IR) systems excel at returning a set of documents relevant to a user's query. However, for many user information needs, a ranked list of documents is less useful than a direct, concise answer to a specific question (e.g., "Who discovered Pluto?"). This paper addresses the challenge of moving from document retrieval to precise answer extraction, aiming to provide users with the exact information they seek, rather than a collection of potentially relevant documents. This is particularly important for questions with a single, factual answer, and can also form the basis for answering quantitative queries.

Methods

The system is built upon two main stages: passage retrieval and linguistic processing.

  • Passage Retrieval: A modified version of the SMART IR system is used to identify relevant documents and then extract passages likely to contain the answer. A passage is defined as a sentence and its two immediate neighbors, allowing for overlapping context.
  • Linguistic Processing: This stage comprises four components:
  • Entity Extraction: A partial parser (Cass) is used to extract specific types of entities like dates, durations, linear measures, and quantities. Specialized code is also employed for extracting proper names.
  • Entity Classification: Extracted entities are classified into categories such as Person, Location, Organization, and Other. This is a crucial step for matching entities to the query's intent.
  • Query Classification: The system determines the category of entity the question is asking for (e.g., a "Who" question seeks a "Person").
  • Entity Ranking: Candidate entities are ranked based on two primary factors: first, whether their classified category matches the query's category, and second, by their frequency and position within the retrieved passages.

Datasets

The system was evaluated on the TREC-8 question-answering track, which provided 198 questions as a blind test set. The evaluation involved human scoring by NIST evaluators. No new datasets were introduced by the authors.

Key Insights and Ideas

  • NLP's Value for Precision: The paper strongly advocates for the utility of Natural Language Processing (NLP) in question answering, demonstrating that it provides significant performance improvements over systems relying solely on IR techniques, especially when precise, short answers are required.
  • Hybrid Approach: The system's strength lies in its hybrid nature, combining the broad coverage of IR with the precision of NLP. This approach offers greater robustness than pure NLP systems and higher precision than pure IR systems.
  • Task Decomposition: Breaking down the problem into independent subtasks (passage retrieval, entity extraction, entity classification, query classification, entity ranking) allows for modular development and evaluation, making the system more manageable and extensible.
  • Answer Length Matters: The results highlight that as answer length constraints become stricter (e.g., 50 bytes vs. 250 bytes), the effectiveness of NLP-based entity extraction becomes more pronounced, outperforming pure passage retrieval.

Key Results

The system achieved a mean score of 0.356 on the TREC-8 evaluation, with correct answers appearing in the top five ranked answers for 46% of the questions. The average answer length produced by the entity-based system was a concise 10.5 bytes.

Comparatively:

  • A 250-byte passage-based system achieved 68% accuracy in the top five and a score of 0.545.
  • A 50-byte passage-based system achieved 38.9% accuracy in the top five and a score of 0.261.

The entity-based system significantly outperformed the 50-byte passage-based system, demonstrating the value of NLP for short answers. The system also showed a tendency to rank correct answers very highly when they were found, with 66% of answers found in the top five being ranked first.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Handling Diverse Entity Types: The system struggled with certain question types, particularly those involving "Other Named Entity" and "Miscellaneous" categories, which accounted for a significant portion of errors. Extending entity extraction to handle arbitrary types (e.g., mountain ranges, films) is a key area for future work.
  • Sophisticated NLP Use: There's potential for improved performance through more advanced NLP techniques, such as utilizing the syntactic context in which entities appear, which is not currently used.
  • Hand-Built System: The system is largely hand-built. The authors suggest that as more features are added, a trainable statistical or machine learning approach would become increasingly desirable, necessitating the development of large, annotated question-answer corpora.
  • Ranking Nuances: Errors in ranking were attributed to factors like over-weighting answers in top-ranked passages, failure to adjust frequencies by "complexity," and not considering linguistic context.

Summary updated

BibTeX
@inproceedings{collins2000answer,
  author = {Michael Collins and Steven P. Abney and Michael Collins and Amit Singhal},
  title = {{Answer Extraction}},
  booktitle = {Anlp},
  year = {2000},
  doi = {10.3115/974147.974188},
}
Figure