Abstract
Standard evaluation metrics like ROUGE and BLEU often fail to distinguish between a correct answer and a semantically opposite one in Machine Reading Comprehension (MRC). This paper introduces simple yet effective adaptations to these metrics by adding "bonus" terms for yes-no opinion agreement and entity list accuracy, significantly improving their correlation with human judgment.
Introduction
In Machine Reading Comprehension (MRC), systems are tasked with reading passages and providing accurate answers. Currently, evaluation relies heavily on lexical overlap metrics such as BLEU (precision-based n-gram overlap) and ROUGE-L (recall-based longest common subsequence). However, these metrics suffer from significant "semantic gaps." For instance, in "Yes-No" questions, a system might output a semantically opposite answer that shares most words with the gold standard, yet receives a high score. Similarly, for "Entity" questions, missing a single critical digit in a date results in a minor penalty despite being a factual failure. As MRC tasks shift toward natural language generation, these limitations make standard metrics increasingly unreliable.
Methodology
The authors propose a "bonus" mechanism to incorporate semantic awareness into BLEU and ROUGE-L without requiring a total overhaul of the metrics.
1. Adaptations on BLEU
The standard BLEU formula is modified by adding bonus terms to the numerator and denominator:
- Yes-No Bonus ($bonus_{yn}$): Awarded if the system correctly predicts the opinion (Yes, No, or Depends).
- Entity Bonus ($bonus_{ent}$): Awarded for correctly identifying entities from a pre-defined gold set.
2. Adaptations on ROUGE-L
For Yes-No questions, a weighted version of the Longest Common Subsequence (LCS) length is added to precision and recall if opinion labels match. For Entity questions, the total length of correctly identified gold entities is added as a bonus to the LCS score.
Experimental Setup
The researchers utilized DuReader, a large-scale Chinese MRC dataset containing a high percentage of yes-no and entity-list questions (36-47%). DuReader includes manual annotations for opinion labels, providing a suitable testbed. A human evaluation set of 500 questions and 2,500 candidate answers from five MRC systems was used to measure correlation with human judgment via the Pearson Correlation Coefficient (PCC).
Results
The adapted metrics showed significant improvements in correlation with human scores. At the single-question level, Adapted ROUGE-L achieved a PCC of 0.570, compared to 0.504 for the vanilla version.
At the system ranking level, Adapted ROUGE-L reached a correlation of 0.792, significantly outperforming the vanilla version's 0.760.
Discussion and Conclusion
The results support the "Crucial Item" hypothesis: in specific question types, certain words carry more information weight than others. By adding simple linear bonuses to transparent metrics, the authors achieved high effectiveness without the complexity of neural evaluators. While these metrics currently depend on gold-standard labels and manual weight tuning, they represent a significant step toward more semantically aware MRC evaluation. Future work will explore adaptations for more complex "description" or "why" questions.
Summary updated