Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

An Yang, Kai Liu, Jing Liu, Yajuan Lyu, Sujian Li

TL;DR

Standard evaluation metrics like ROUGE and BLEU often fail to distinguish between a correct answer and a semantically opposite one in Machine Reading Comprehension (MRC). This paper introduces simple yet effective adaptations to these metrics by adding "bonus" terms for yes-no opinion agreement and entity list accuracy, significantly improving their correlation with human judgment.

What Problem Does This Paper Address?

In the world of Machine Reading Comprehension (MRC), we want AI to read a passage and answer questions accurately. Currently, we evaluate these systems using metrics like BLEU (precision-based n-gram overlap) and ROUGE-L (recall-based longest common subsequence).

The problem is that these metrics rely almost entirely on lexical overlap—how many words the machine's answer shares with the human's "gold" answer. This creates two major blind spots:

  1. Yes-No Questions: An answer like "The router has an impact" and "The router has no impact" share almost all the same words, resulting in a high ROUGE/BLEU score, even though the answers are diametrically opposed.
  2. Entity Questions: If a question asks for a list of dates or names, missing a single digit or one name in a list is a catastrophic failure in accuracy but only a tiny penalty in standard overlap metrics.

As MRC tasks move toward generating natural language answers rather than just extracting spans of text, these "semantic gaps" make standard metrics increasingly unreliable.

Methods

The authors propose a "bonus" mechanism to bake semantic awareness into the existing math of BLEU and ROUGE-L.

1. Adaptations on BLEU

The standard BLEU formula calculates precision ($P_i$) for n-grams. The authors modify this by adding a bonus term to both the numerator and denominator:

  • Yes-No Bonus ($bonus_{yn}$): If the system correctly predicts the opinion (Yes, No, or Depends), it receives a bonus based on the n-gram overlap.
  • Entity Bonus ($bonus_{ent}$): If the system correctly identifies entities (like "230 BC" or "ten years") from a pre-defined gold entity set, it gets a bonus proportional to those hits.

2. Adaptations on ROUGE-L

ROUGE-L measures the Longest Common Subsequence (LCS). The authors apply a similar logic:

  • For Yes-No questions, if the opinion labels match, they add a weighted version of the LCS length to the precision and recall calculations.
  • For Entity questions, they add the total length of correctly identified gold entities as a bonus to the LCS score.

Crucially, these adaptations don't require a total overhaul of the metrics; they simply "nudge" the scores higher when the system gets the "crucial" parts of the answer right.

Datasets

The researchers primarily used DuReader, a large-scale Chinese MRC dataset from Baidu.

  • Why DuReader? Unlike SQuAD (which is mostly factoid-based), DuReader contains a high percentage of yes-no and entity-list questions (roughly 36-47%). It also already includes manual annotations for opinion labels (Yes/No/Depends), making it the perfect testbed for these adapted metrics.
  • Human Evaluation Set: They sampled 500 questions (201 yes-no, 201 entity, 98 description) and collected 2,500 candidate answers from five different MRC systems to create a benchmark for human-metric correlation.

Key Insights and Ideas

  • The "Crucial Item" Hypothesis: In specific question types, certain words (like "no" or a specific date) carry significantly more "information weight" than others. Standard metrics treat all words as equal; these adaptations fix that.
  • Simplicity Scales: Instead of building a complex neural-network-based evaluator (which can be a "black box"), the authors showed that adding a simple linear bonus to trusted, transparent metrics like BLEU/ROUGE is highly effective.
  • Opinion Labels are Cheap: The authors argue that while annotating full answers is hard, adding a simple "Yes/No" label to reference answers during dataset construction is low-cost but provides high value for evaluation.

Key Results

The effectiveness was measured using the Pearson Correlation Coefficient (PCC), which tracks how closely the automatic metric follows human scores (1.0 is a perfect match).

  • Overall Performance: On a single-question level, Adapted ROUGE-L achieved a PCC of 0.570, compared to 0.504 for vanilla ROUGE-L.
  • Entity Questions: The improvement was even more dramatic for entity-based questions, where Adapted BLEU-4 jumped from 0.397 to 0.469.
  • System Ranking: When looking at the overall score of an entire MRC system (the "overall score level"), Adapted ROUGE-L reached a correlation of 0.792, significantly outperforming the vanilla version (0.760).

Limitations and Future Work

  • Dependency on Labels: The adapted metrics require gold-standard opinion labels and entity lists. While DuReader provides these, many older or smaller datasets do not.
  • Weight Tuning: The "bonus weights" ($\alpha$ and $\beta$) currently need to be set manually. The authors noted that as these weights increase, correlation generally improves, but finding the "optimal" weight for every scenario remains an open task.
  • Scope: The paper focuses on yes-no and entity questions. Future work is needed to adapt metrics for "description" or "why" questions, which are even more linguistically complex.

Summary updated

Abstract

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.
BibTeX
@inproceedings{yang2018adaptations,
  author = {An Yang and Kai Liu and Jing Liu and Yajuan Lyu and Sujian Li},
  title = {{Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task}},
  booktitle = {QA @ ACL},
  year = {2018},
  url = {https://openreview.net/forum?id=H1ZC2nxdWr},
}

Loading related content...

Figure