A Probabilistic Rasch Analysis of Question Answering Evaluations

Rense Lange, Juan Moran, Warren R. Greiff, Lisa FerroNAACL 2004

TL;DR

This paper introduces the Rasch measurement model, a probabilistic approach from psychometrics, to evaluate Question Answering (QA) systems. It demonstrates that this method provides a more robust and theoretically grounded way to measure both system ability and question difficulty on a common scale, overcoming limitations of traditional sum-score evaluations.

What Problem Does This Paper Address?

Evaluating the performance of computational systems on complex tasks like Question Answering (QA) has traditionally relied on simple metrics like the number of correct answers or percentages. While intuitive, these sum-score approaches often fail to build a cumulative body of knowledge, making it difficult to form theories or make precise comparisons. This is because they don't adequately account for the inherent difficulty of the questions themselves and how that difficulty interacts with a system's ability. The paper argues that this is a fundamental issue in Human Language Technology (HLT) evaluation, analogous to challenges faced in educational testing for decades. The core problem is the lack of a common, invariant metric that can reliably compare systems across different evaluations or question sets over time.

Methods

The paper proposes and applies the Rasch model for binary data. This probabilistic model, originating in psychometrics, posits that the probability of a correct response to a question by a system depends solely on the difference between the system's ability and the question's difficulty.

Key aspects of the Rasch model application include:

  • Decomposition of Performance: The model separates performance into two additive components: system ability ($S_s$) and question difficulty ($Q_q$). This is expressed on a common scale (logits) where $P_{sq} = (1+e^{Q_q-S_s})^{-1}$.
  • Parameter Estimation: Maximum likelihood estimation is used to derive estimates for system abilities and question difficulties, along with their standard errors.
  • Fit Analysis: The paper utilizes "Outfit" statistics to assess how well individual questions and systems fit the Rasch model's assumptions. Misfitting items or systems indicate unexpected response patterns, suggesting they might be testing different facets or behaving unusually.
  • Test Equating: A crucial application demonstrated is "equating," where different sets of questions (e.g., from different years) can be placed on the same scale. This is achieved by including a common set of "equating questions" that link the difficulty scales of different test administrations.

Datasets

The primary dataset used is the Question Answering (QA) track of the 2002 Text Retrieval Conference (TREC) competition. This dataset consisted of responses from 67 systems to 500 questions. The paper notes that NIST judges assessed answers as correct, incorrect, non-exact, or unsupported, and for this analysis, the last three categories were treated as "incorrect" to create binary (correct/incorrect) responses. Ten questions were excluded as no system answered them correctly.

Key Insights and Ideas

  • A Unified Scale for Ability and Difficulty: The Rasch model provides a single, invariant scale (logits) to measure both how difficult a question is and how capable a system is. This is a significant improvement over raw scores, which are confounded by question difficulty.
  • Diagnosing System and Question Behavior: The "Outfit" statistics allow researchers to identify questions that behave unexpectedly (e.g., are answered by low-ability systems but not high-ability ones) and systems that perform unusually relative to the overall patterns. This offers a powerful diagnostic tool.
  • Robustness of Equating: The simulation study demonstrates that even with maximally different sets of questions (easy vs. hard), Rasch equating can produce highly correlated system ability estimates, suggesting that systems can be reliably compared across different test administrations.
  • Limitations of Raw Scores: The paper highlights that raw scores (and percentages) are not only dependent on the specific questions used but also exhibit non-linear relationships when comparing systems across different question sets, making direct comparisons problematic.

Key Results

  • Satisfactory Model Fit: The Rasch model was applied to the TREC 2002 QA data, and satisfactory model fit was obtained, indicating the model's applicability to this domain.
  • Question Difficulty Range: Question difficulties ranged from approximately -3 to +3 logits, with most systems' abilities falling in a lower range (around -1.98 logits), suggesting that many questions were quite difficult for the participating systems.
  • Equating Effectiveness: Simulations showed that using as few as 20-50 equating questions could yield high correlations (r=0.90 to 0.94) between system abilities estimated from different question sets. This indicates that a test-invariant metric can be defined.
  • Raw Score vs. Rasch Equating: Scatter plots revealed that Rasch-based ability estimates from "easy" and "hard" question sets were highly correlated (r=0.94), while raw score comparisons showed a lower correlation (r=0.82) and a significant non-linear trend.

Limitations and Future Work

  • Misfitting Questions/Systems: While the paper shows that removing misfitting questions had a negligible impact on overall system estimates (r=0.99 correlation), the authors acknowledge that identifying and analyzing these misfitting items is crucial for deeper understanding and theory development.
  • Further Model Exploration: The paper suggests exploring more complex Rasch models, such as the 2-parameter IRT model, which can estimate a discrimination parameter in addition to difficulty. This could provide more nuanced information about question characteristics.
  • Broader HLT Applications: The authors propose that Rasch modeling can be extended to other HLT evaluation contexts beyond QA, including areas with multi-level responses, counts, proportions, or rater effects, and metrics like average precision, R-precision, and F-measure.
  • Test Equating for Future TREC: The paper strongly recommends introducing test equating into formal HLT evaluations, specifically suggesting that NIST reuse a set of questions annually for calibration to ensure comparability across TREC competitions.

Summary updated

BibTeX
@inproceedings{lange2004probabilistic,
  author = {Rense Lange and Juan Moran and Warren R. Greiff and Lisa Ferro},
  title = {{A Probabilistic Rasch Analysis of Question Answering Evaluations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure