Is Question Answering Better than Information Retrieval? Towards a Task-Based Evaluation Framework for Question Series

Jimmy J. LinNAACL 2007 · OpenReview

Abstract

This paper introduces a novel evaluation framework for question series and employs it to explore the effectiveness of QA and IR systems at addressing users’ information needs. The framework is based on the notion of recall curves, which characterize the amount of relevant information contained within a fixed-length text segment. Although it is widely assumed that QA technology provides more efficient access to information than IR systems, our experiments show that a simple IR baseline is quite competitive. These results help us better understand the role of NLP technology in QA systems and suggest directions for future research.

Introduction

In the mid-2000s, Question Answering (QA) research aimed to evolve beyond simple document retrieval to providing precise answers. However, a gap emerged between system evaluation and user needs. Users generally prefer answers embedded in context rather than isolated strings. Furthermore, information needs often manifest as "question series" centered on a specific target. This paper investigates whether the complex natural language processing (NLP) machinery of QA systems truly outperforms traditional Information Retrieval (IR) when context is provided.

Methodology

The study introduces a task-based evaluation framework centered on recall curves. This metric measures "nugget recall"—the fraction of relevant facts (nuggets) captured—as a function of the total response length in non-whitespace characters. This approach simulates a user reading through a response, assessing how much information is gained per character read.

To establish a baseline, a simple IR system was built using Lucene. For any given topic, the system retrieved the top 100 documents, extracted sentences containing target keywords, and ranked them. This baseline was compared against official runs from the TREC 2004 and 2005 QA tracks. To ensure a fair comparison, "exact" answers from QA systems were projected back into their original source sentences to provide the context users prefer.

Results and Analysis

Factoid Questions

Evaluation on factoid questions—those seeking specific data points—showed that while the top-tier QA systems outperformed the IR baseline, the advantage was not universal.

Recall curves for TREC 2004 and 2005 factoid questions.

As shown in the recall curves, the "median" TREC system often performed no better than the simple Lucene IR baseline, particularly as response length increased. The value of NLP technology appears most significant in the middle range of response lengths, where it can pinpoint information faster than raw retrieval.

"Other" Questions

The results for "other" questions—open-ended requests for information about a target—were even more striking. These questions essentially require mini-summaries.

Recall curves for TREC 2004 and 2005 "other" questions.

In the 2004 dataset, the simple IR baseline outperformed almost all specialized QA systems at higher recall levels. By 2005, the performance of all systems, including the IR baseline, was so tightly clustered that the sophisticated QA machinery was nearly indistinguishable from basic keyword-based sentence extraction.

Conclusion

The findings challenge the assumption of QA's inherent superiority over IR. While high-end NLP provides efficiency in pinpointing facts, a simple IR baseline remains remarkably competitive, especially for open-ended queries. Future research should pivot from mere "pinpointing" toward answer generation and synthesis, focusing on creating coherent, fluent summaries that address complex information needs more effectively than extracted sentence lists.

Summary updated

Abstract

This paper introduces a novel evaluation framework for question series and employs it to explore the effectiveness of QA and IR systems at addressing users’ information needs. The framework is based on the notion of recall curves, which characterize the amount of relevant information contained within a fixed-length text segment. Although it is widely assumed that QA technology provides more efficient access to information than IR systems, our experiments show that a simple IR baseline is quite competitive. These results help us better understand the role of NLP technology in QA systems and suggest directions for future research.
BibTeX
@inproceedings{lin2007is,
  author = {Jimmy J. Lin},
  title = {{Is Question Answering Better than Information Retrieval? Towards a Task-Based Evaluation Framework for Question Series}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
  url = {https://openreview.net/forum?id=H1EzG7W_ZH},
}

Loading related content...

Figure