TL;DR
This paper introduces "predictive annotation," a novel technique to improve natural language question answering by enriching both queries and text with semantic category labels (QA-Tokens). This allows a system to better match question intent with potential answer phrases, leading to improved ranking of suspected answers, as demonstrated in the TREC Q&A evaluations.
What Problem Does This Paper Address?
Traditional Information Retrieval (IR) systems struggle with the semantic nuances of natural language questions. They often treat queries as bags of words, failing to understand the underlying intent (e.g., seeking a location, person, or date). Conversely, full Natural Language Processing (NLP) can be computationally expensive. The authors aim to bridge this gap by developing a system that leverages the efficiency of IR while incorporating shallow NLP techniques to better understand question semantics and identify relevant answer phrases within retrieved documents.
Methods
The core innovation is predictive annotation. This technique involves:
- Query Augmentation: Queries are processed against a set of templates that can replace words with semantic category markers (QA-Tokens). For example, "Where" might be replaced with "PLACE$".
- Text Annotation: Documents are pre-processed using a tool called Textract (for lemmatization and named entity recognition) and then a new module, Resporator, annotates text segments with QA-Tokens based on pattern matching. For instance, "for 5 centuries" might be tagged with "DURATION$".
- IR System (GuruQA): This system scores text passages (not just documents) using a weighting scheme where QA-Tokens are heavily weighted, followed by proper nouns and other words. It returns a list of matching passages.
- Answer Selection (AnSel/Werlect): Two algorithms, AnSel and Werlect, then rank potential answer spans within these passages.
- AnSel: Uses a linear combination of seven predictive features (e.g., span position, proximity to query terms, span type) trained via logistic regression to assign a total score to each span.
- Werlect: Employs a two-step, rule-based approach that considers similar features but also incorporates frequency of terms across passages and a modified "notinq" feature.
Datasets
- TREC Corpus: A 2GB collection of articles from four news agencies was used for both training and evaluation.
- TREC Q&A Evaluation Data:
- TR38: A training set of 38 questions with provided answers.
- T200: A test set of 200 questions used for evaluation.
Key Insights and Ideas
- Semantic Tagging is Key: Augmenting both queries and text with semantic labels (QA-Tokens) significantly improves the ability to match question intent with potential answer phrases, going beyond simple keyword matching.
- Phrase-Level Answers: Fact-seeking questions are often answered by specific phrases (e.g., "President Clinton," "New York City"), and the system is designed to identify and rank these phrases.
- Feature Engineering Matters: The success of AnSel and Werlect hinges on carefully selected features that capture the likelihood of a text span being a correct answer. Features like the presence of specific QA-Tokens, proximity to query terms, and the type of phrase are crucial.
- Different Ranking Strategies: Both AnSel (linear combination, learned weights) and Werlect (rule-based, two-step) achieve competitive results, suggesting that multiple approaches can be effective for answer ranking. Werlect showed an advantage on questions that couldn't be classified with a QA-Token.
Key Results
- The system participated in the official TREC Q&A evaluation, providing 50-byte and 250-byte extracts for 200 questions.
- AnSel (runs A50 and A25) performed marginally better overall than Werlect.
- A50 achieved a cumulative RAR of 63.22 on T200, answering 49 questions correctly on the first try and 39 more within the top five answers.
- A250 achieved a cumulative RAR of 85.17, answering 71 questions correctly on the first try and 38 more within the top five.
- In comparison to other TREC submissions, AnSel (50-byte output) scored 0.430, and Werlect (50-byte output) scored 0.395.
Limitations and Future Work
- QA-Token Classification: Predictive annotation works best for certain question types (e.g., Where, When, What, Which, How+adjective). Questions like "Why" are harder to classify and answer with phrases.
- Unclassified Questions: Werlect showed an advantage on questions that could not be classified with a QA-Token, suggesting that robustness to unclassified queries is an area for improvement.
- Future Work: The authors suggest that a system based on predictive annotation can deliver very good results and imply that further refinement of features and algorithms could lead to even better performance. The internal output of the answer selection module (ranked spans) could be used in applications beyond TREC's external extract requirements.
Summary updated