Ranking suspected answers to natural language questions using predictive annotation

Dragomir R. Radev, John M. Prager, Valerie Samn

TL;DR

This paper introduces "predictive annotation," a novel technique that augments both queries and text with semantic category labels (QA-Tokens) to improve the ranking of potential answers to natural language questions. By bridging the gap between traditional Information Retrieval and computationally expensive Natural Language Processing, their system achieves competitive results in the TREC Q&A evaluations.

What Problem Does This Paper Address?

Traditional Information Retrieval (IR) systems excel at efficiently searching vast amounts of text but struggle with understanding the semantic meaning of queries and documents. Conversely, Natural Language Processing (NLP) can grasp semantics but is often too computationally intensive for large-scale question answering. The authors aim to find a middle ground by enhancing a modified IR system with shallow NLP parsing. A key challenge they highlight is how standard IR systems, which treat queries as bags of words, fail when stop words like "where" are removed, losing the crucial semantic intent (e.g., seeking a location). Without this semantic understanding, the system cannot effectively identify relevant passages.

Methods

The core innovation is predictive annotation. This technique involves:

  • Query Augmentation: Queries are processed to replace certain words with semantic category markers called QA-Tokens. For instance, "Where" might be replaced by "PLACE$". This helps the search engine understand the type of information being sought.
  • Text Annotation: Documents are pre-processed to identify and label recognized entities (like places, names, dates) with corresponding QA-Tokens. This is done using a combination of tools like Textract (for lemmatization and named entity recognition) and a custom module called Resporator, which uses pattern matching to assign QA-Tokens.
  • Modified IR System (GuruQA): The system then uses a traditional bag-of-words matching approach, but with the added benefit of QA-Tokens. GuruQA scores text passages rather than entire documents, using a weighting scheme where QA-Tokens are highly valued.
  • Answer Selection (AnSel/Werlect): After GuruQA returns a set of matching passages, two algorithms, AnSel and Werlect, are used to rank the potential answer spans within these passages.
  • AnSel: Employs a linear combination of seven predictive features (span position, passage position, span class count, words not in query, span type position, distance to query words, and passage relevance score) learned via logistic regression.
  • Werlect: Uses a two-step, rule-based process, incorporating features like span type, average distance, passage relevance, and frequency of occurrence.

Datasets

  • TREC Corpus: A 2-GB collection of articles from four news agencies was used for both training and evaluation.
  • Training Set (TR38): 38 questions with NIST-provided answers were used to train the system.
  • Test Set (T200): 200 questions were used for evaluation.

The paper does not introduce new benchmarks but evaluates its system on the established TREC Q&A evaluation framework.

Key Insights and Ideas

  • Semantic Intent is Crucial: Simply matching keywords is insufficient for question answering. Understanding the type of information requested (location, person, date) is paramount.
  • Bridging IR and NLP: Predictive annotation offers a practical way to inject semantic understanding into IR systems without the full computational cost of deep NLP.
  • Feature Engineering for Ranking: The success of AnSel and Werlect hinges on identifying features that strongly correlate with the correctness of a potential answer span. The paper demonstrates that span position, type, and relevance scores are particularly important.
  • Passage-Level Scoring: Scoring passages instead of entire documents allows for more focused retrieval and a better chance of isolating the answer.

Key Results

The system participated in the official TREC Q&A evaluation, providing 50-byte and 250-byte extracts for 200 questions.

  • Overall Performance: AnSel generally performed slightly better than Werlect.
  • TREC Comparison: The system achieved competitive results against other participants. For example, the Ansel 50-byte output achieved an MRAR of 0.430, and Werlect achieved 0.395. On 250-byte output, AnSel scored 0.319 and Werlect 0.280.
  • Unclassified Questions: Werlect showed a notable advantage (MRAR of 3.5 vs. AnSel's 2.0) on questions that could not be classified with a QA-Token, suggesting its rule-based approach is more robust in such cases.

Limitations and Future Work

  • Limitations:
    • Predictive annotation is noted to work best for "Where," "When," "What," "Which," and "How+adjective" questions, being less effective for "Why" and "How+verb" questions, which are often answered by explanations rather than simple phrases.
  • The system's performance on questions that could not be classified with a QA-Token was an area where Werlect showed improvement but still had room for growth.
  • Future Work: The paper suggests that a system based on predictive annotation can deliver very good results and implies that further refinement of the annotation and ranking techniques could lead to even better performance. The authors also mention that the internal output of their system (ranked spans) could be useful for applications beyond TREC's external output format.

Summary updated

BibTeX
@inproceedings{radev2000ranking,
  author = {Dragomir R. Radev and John M. Prager and Valerie Samn},
  title = {{Ranking suspected answers to natural language questions using predictive annotation}},
  booktitle = {ANLP},
  year = {2000},
  url = {https://openreview.net/forum?id=j7w0pc7wNo},
}
Figure