Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

An Yang, Kai Liu, Jing Liu, Yajuan Lyu, Sujian Li

Abstract

Standard evaluation metrics like ROUGE and BLEU often fail to distinguish between a correct answer and a semantically opposite one in Machine Reading Comprehension (MRC). This paper introduces simple yet effective adaptations to these metrics by adding "bonus" terms for yes-no opinion agreement and entity list accuracy, significantly improving their correlation with human judgment.

Introduction

In Machine Reading Comprehension (MRC), systems are tasked with reading passages and providing accurate answers. Currently, evaluation relies heavily on lexical overlap metrics such as BLEU (precision-based n-gram overlap) and ROUGE-L (recall-based longest common subsequence). However, these metrics suffer from significant "semantic gaps." For instance, in "Yes-No" questions, a system might output a semantically opposite answer that shares most words with the gold standard, yet receives a high score. Similarly, for "Entity" questions, missing a single critical digit in a date results in a minor penalty despite being a factual failure. As MRC tasks shift toward natural language generation, these limitations make standard metrics increasingly unreliable.

Methodology

The authors propose a "bonus" mechanism to incorporate semantic awareness into BLEU and ROUGE-L without requiring a total overhaul of the metrics.

1. Adaptations on BLEU

The standard BLEU formula is modified by adding bonus terms to the numerator and denominator:

  • Yes-No Bonus ($bonus_{yn}$): Awarded if the system correctly predicts the opinion (Yes, No, or Depends).
  • Entity Bonus ($bonus_{ent}$): Awarded for correctly identifying entities from a pre-defined gold set.

2. Adaptations on ROUGE-L

For Yes-No questions, a weighted version of the Longest Common Subsequence (LCS) length is added to precision and recall if opinion labels match. For Entity questions, the total length of correctly identified gold entities is added as a bonus to the LCS score.

Experimental Setup

The researchers utilized DuReader, a large-scale Chinese MRC dataset containing a high percentage of yes-no and entity-list questions (36-47%). DuReader includes manual annotations for opinion labels, providing a suitable testbed. A human evaluation set of 500 questions and 2,500 candidate answers from five MRC systems was used to measure correlation with human judgment via the Pearson Correlation Coefficient (PCC).

Results

The adapted metrics showed significant improvements in correlation with human scores. At the single-question level, Adapted ROUGE-L achieved a PCC of 0.570, compared to 0.504 for the vanilla version.

Pearson Correlation Coefficient (PCC) at the single-question level.

At the system ranking level, Adapted ROUGE-L reached a correlation of 0.792, significantly outperforming the vanilla version's 0.760.

Pearson Correlation Coefficient (PCC) at the system ranking level.

Discussion and Conclusion

The results support the "Crucial Item" hypothesis: in specific question types, certain words carry more information weight than others. By adding simple linear bonuses to transparent metrics, the authors achieved high effectiveness without the complexity of neural evaluators. While these metrics currently depend on gold-standard labels and manual weight tuning, they represent a significant step toward more semantically aware MRC evaluation. Future work will explore adaptations for more complex "description" or "why" questions.

Summary updated

Abstract

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.
BibTeX
@inproceedings{yang2018adaptations,
  author = {An Yang and Kai Liu and Jing Liu and Yajuan Lyu and Sujian Li},
  title = {{Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task}},
  booktitle = {QA @ ACL},
  year = {2018},
  url = {https://openreview.net/forum?id=H1ZC2nxdWr},
}
Figure