Evaluating the Evaluation: A Case Study Using the TREC 2002 Question Answering Track

Ellen M. VoorheesNAACL 2003

TL;DR

This paper critically examines the TREC 2002 Question Answering (QA) track evaluation methodology. It finds that despite changes to the task, the evaluation remains stable and reliable for comparing QA systems, and it empirically quantifies the minimum score difference needed to confidently declare one system superior to another.

What Problem Does This Paper Address?

Evaluating complex AI systems like Question Answering (QA) is crucial for driving progress and technology transfer. However, poorly designed evaluations can waste valuable research effort and lead to misleading conclusions. This paper addresses the need to validate new evaluation tasks, specifically the TREC 2002 QA track, to ensure its reliability and understand the confidence we can place in its results. The TREC QA track has evolved, and this study aims to confirm that the new task definition still provides a robust and meaningful comparison of system performance.

Methods

The authors analyze the TREC 2002 QA track's evaluation by:

  • Assessing Human Judgment Reliability: They investigate how consistently human assessors could judge the "exactness" of answers, a key change in the 2002 task. This involved analyzing disagreements among three independent assessors for each response.
  • Evaluating Ranking Stability: They examine if the system rankings remain stable even with variations in human judgments. This was done by comparing rankings derived from individual assessor judgments against an "adjudicated" (final) judgment set, using Kendall's tau correlation.
  • Performing Sensitivity Analysis: They empirically determine the minimum score difference required between two systems to confidently conclude that one is better than the other. This involved simulating evaluations with varying numbers of questions and analyzing the "swap rate" (when rankings disagree).

Datasets

  • TREC 2002 QA Track Data: This includes a large corpus of newswire documents (from AP, New York Times, and Xinhua) and a set of 500 closed-class questions.
  • MSNSearch and AskJeeves Logs: These logs were used to source candidate questions for the test set.

The 2002 task introduced a new benchmark by requiring "exact answers" and a single, ranked response per question, differing from previous TREC QA tracks that allowed text snippets and multiple responses.

Key Insights and Ideas

  • Human Judgment on "Exactness" is Generally Consistent: While disagreements exist, they primarily stem from subtle differences in interpreting "exactness" (e.g., whether "Kidman" is as exact as "actress Nicole Kidman") rather than fundamental misunderstandings of the task. These differences are similar to those observed in earlier TREC QA tracks.
  • The 2002 QA Task is Stable: Despite the new task requirements (exact answers, single ranked response), the system rankings remain acceptably stable across different assessor judgment sets. This indicates the evaluation methodology is robust.
  • Confidence-Weighted Score is Sensitive: The confidence-weighted score, which prioritizes correctly ranked answers, is more sensitive to judgment differences than a simple count of correct answers. This means smaller score differences can be meaningful, but also that rankings might be slightly less stable.
  • Quantifying "Meaningful" Differences is Crucial: Not all score differences are statistically significant. The paper provides a critical insight into how to determine the minimum score difference needed to be confident that one system is truly better than another, which is essential for practical evaluation.

Key Results

  • Assessor Disagreements: 11.8% of response pairs showed some disagreement among three assessors. The most common disagreements were between "right" and "inexact" judgments.
  • Ranking Stability: Kendall's tau correlations between rankings derived from different judgment sets were high, averaging 0.930 for the confidence-weighted score and 0.945 for the raw count of correct answers. This is considered sufficiently stable for evaluation.
  • Minimum Score Difference for Confidence: For the TREC 2002 QA track's test set of 500 questions, a minimum difference of 0.07 in confidence-weighted scores is required to achieve a less than 5% error rate in concluding one system is better than another.

Limitations and Future Work

  • Reusable Infrastructure: The paper notes that the TREC 2002 QA task, like previous ones, does not produce a truly reusable QA evaluation infrastructure. It's difficult to use the existing judgment sets to evaluate new runs on the same data. Future work should focus on developing such reusable infrastructure.
  • Approximation Methods: While methods exist to approximate evaluation scores for new runs, they are not completely reliable.
  • Sensitivity to Question Set Size: The sensitivity analysis highlights the impact of question set size on evaluation reliability. While the paper extrapolates to 500 questions, further research could explore the optimal size for different evaluation goals.

Summary updated

BibTeX
@inproceedings{voorhees2003evaluating,
  author = {Ellen M. Voorhees},
  title = {{Evaluating the Evaluation: A Case Study Using the TREC 2002 Question Answering Track}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure