Using Question Series to Evaluate Question Answering System Effectiveness

Ellen VoorheesEMNLP 2005

TL;DR

This paper introduces and validates the use of "question series" as a more effective evaluation unit for question answering (QA) systems than individual questions. By modeling user interactions and aggregating scores at this series level, the TREC QA track achieved a more reliable and meaningful evaluation, better reflecting real-world dialogue scenarios.

What Problem Does This Paper Address?

Evaluating question answering (QA) systems has traditionally relied on assessing their performance on individual questions. However, this approach has limitations. For instance, individual questions can lead to highly skewed score distributions, making it difficult to reliably compare systems. Furthermore, it doesn't fully capture the nuances of how users interact with QA systems in a conversational or information-seeking context, where questions often build upon each other. The TREC QA track, aiming to model aspects of dialogue processing, needed a more robust evaluation methodology that could handle different question types and provide a more stable measure of system effectiveness.

Methods

The core innovation presented is the use of question series as the primary unit of evaluation. A question series is a collection of related questions, all centered around a common "target" (a person, organization, or thing). These series were designed to mimic aspects of information-seeking dialogues.

Key methodological aspects include:

  • Series Structure: Each series typically included factoid questions (short, factual answers), list questions (multiple instances of information), and a final "Other" question (asking for interesting, unprompted information).
  • Contextual Processing: While systems were required to process series in order, allowing them to use information from earlier questions to answer later ones, the evaluation itself was designed to be "per-series." This means the score for a series was computed independently, regardless of how the system internally processed the questions.
  • Scoring: Individual question types (factoid, list, Other) have specific scoring protocols. Factoid questions are scored with binary correctness. List questions use instance precision and recall, combined with an F-measure. "Other" questions are evaluated using a more complex "nugget" based approach, where information is broken down into atomic pieces and assessed for recall and precision.
  • Aggregation: The paper argues for aggregating scores at the series level, rather than averaging scores by question type. This series-level aggregation is shown to be more reliable.
  • Reliability Analysis: A statistical methodology, adapted from document retrieval evaluations, was used to empirically estimate the reliability of the per-series evaluation. This involved analyzing how often different test set sizes and score differences lead to correct conclusions about system effectiveness.

Datasets

The primary dataset used is the TREC 2004 Question Answering (QA) track test set. This set comprised 65 question series, each with a target entity. The series contained a total of 230 factoid questions, 56 list questions, and 65 "Other" questions. The paper notes that these series were edited versions of those collected from actual information seekers to fit evaluation constraints.

Key Insights and Ideas

  • Series as a Meaningful Granularity: Question series provide a level of granularity that is both meaningful at the task level (representing a single user interaction) and large enough to avoid the highly skewed score distributions seen with individual questions.
  • Dialogue Modeling: The series structure effectively models aspects of dialogue processing, allowing for the investigation of contextualized questions and anaphora resolution (how objects are referred to).
  • Improved Evaluation Reliability: Aggregating scores by series, rather than by question type, leads to a more reliable evaluation. This is because series score distributions are less skewed and more closely resemble per-topic scores in document retrieval, allowing established reliability analysis methods to be applied effectively.
  • Series Scores are More Informative: Box-and-whisker plots show that series scores are generally less skewed towards zero compared to individual question scores, making it easier to discern meaningful differences between systems.
  • Practical Implications for System Comparison: The reliability analysis demonstrates that a relatively small difference in per-series scores (e.g., 0.05) can be statistically significant with a test set of 64 series, providing confidence in system comparisons.

Key Results

  • The TREC 2004 QA track used 65 question series, with 64 series ultimately used for evaluation due to one being unjudged.
  • An analysis of the TREC 2004 QA track per-series evaluation showed it to be reliable for differences in scores often observed in practice.
  • The Kendall τ correlation between rankings derived from question-type aggregation and series aggregation was 0.971, indicating high agreement, though absolute scores could differ.
  • For the TREC 2004 test set, a significant percentage of individual questions had zero median scores (92.2% for factoid, 70.9% for list, 64.1% for Other), highlighting the skewness problem with individual question evaluation. In contrast, only a few series (S21, S25, S37, S39) had median scores of 0.0.
  • The reliability analysis suggests that a score difference of at least 0.05 is required for a 5% error rate with 64 series in the test set.

Limitations and Future Work

  • Edited Series: The question series used in TREC were "heavily edited versions" of those collected from information seekers to meet evaluation requirements. This means they might not fully represent the "stilted conversational style" or the "full range of information requested" in real dialogues.
  • System Processing Independence: While the series structure encourages contextual processing, systems were allowed to process questions independently. The evaluation focused on the per-series outcome, not the internal processing strategy.
  • Future Work: The paper implies that further investigation into the optimal weighting of different question types within series or exploring more sophisticated dialogue modeling within evaluations could be beneficial. The methodology for reliability analysis, originally developed for document retrieval, was successfully applied here, suggesting its broader applicability.

Summary updated

BibTeX
@inproceedings{voorhees2005using,
  author = {Ellen Voorhees},
  title = {{Using Question Series to Evaluate Question Answering System Effectiveness}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure