Abstract
This paper introduces a novel evaluation framework for question series and employs it to explore the effectiveness of QA and IR systems at addressing users’ information needs. The framework is based on the notion of recall curves, which characterize the amount of relevant information contained within a fixed-length text segment. Although it is widely assumed that QA technology provides more efficient access to information than IR systems, our experiments show that a simple IR baseline is quite competitive. These results help us better understand the role of NLP technology in QA systems and suggest directions for future research.
Introduction
In the mid-2000s, Question Answering (QA) research aimed to evolve beyond simple document retrieval to providing precise answers. However, a gap emerged between system evaluation and user needs. Users generally prefer answers embedded in context rather than isolated strings. Furthermore, information needs often manifest as "question series" centered on a specific target. This paper investigates whether the complex natural language processing (NLP) machinery of QA systems truly outperforms traditional Information Retrieval (IR) when context is provided.
Methodology
The study introduces a task-based evaluation framework centered on recall curves. This metric measures "nugget recall"—the fraction of relevant facts (nuggets) captured—as a function of the total response length in non-whitespace characters. This approach simulates a user reading through a response, assessing how much information is gained per character read.
To establish a baseline, a simple IR system was built using Lucene. For any given topic, the system retrieved the top 100 documents, extracted sentences containing target keywords, and ranked them. This baseline was compared against official runs from the TREC 2004 and 2005 QA tracks. To ensure a fair comparison, "exact" answers from QA systems were projected back into their original source sentences to provide the context users prefer.
Results and Analysis
Factoid Questions
Evaluation on factoid questions—those seeking specific data points—showed that while the top-tier QA systems outperformed the IR baseline, the advantage was not universal.
As shown in the recall curves, the "median" TREC system often performed no better than the simple Lucene IR baseline, particularly as response length increased. The value of NLP technology appears most significant in the middle range of response lengths, where it can pinpoint information faster than raw retrieval.
"Other" Questions
The results for "other" questions—open-ended requests for information about a target—were even more striking. These questions essentially require mini-summaries.
In the 2004 dataset, the simple IR baseline outperformed almost all specialized QA systems at higher recall levels. By 2005, the performance of all systems, including the IR baseline, was so tightly clustered that the sophisticated QA machinery was nearly indistinguishable from basic keyword-based sentence extraction.
Conclusion
The findings challenge the assumption of QA's inherent superiority over IR. While high-end NLP provides efficiency in pinpointing facts, a simple IR baseline remains remarkably competitive, especially for open-ended queries. Future research should pivot from mere "pinpointing" toward answer generation and synthesis, focusing on creating coherent, fluent summaries that address complex information needs more effectively than extracted sentence lists.
Summary updated