MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Introduction

Evaluating generative reading comprehension (RC) is a significant challenge in NLP. Unlike span-selection tasks, where a model extracts a specific text segment, generative RC requires models to produce free-text answers. Standard metrics like BLEU or ROUGE, which rely on n-gram overlap between a candidate and a reference answer, often fail to capture semantic correctness. They ignore the context provided by the source passage and can be misled by minor lexical changes that fundamentally alter meaning.

To address this, we introduce MOCHA (MOdeling Correctness with Human Annotations), a large-scale dataset of human judgments on RC model outputs, and LERC (Learned Evaluation metric for Reading Comprehension), a context-aware metric designed to "read" the passage and question alongside the answers.

The MOCHA Dataset

MOCHA consists of 40,000 human-annotated instances across six diverse QA datasets, including NarrativeQA, MCScript, CosmosQA, and SocialIQA, as well as span-based datasets like DROP and Quoref. Human annotators scored model-generated candidate answers on a scale of 1 to 5 based on how well they captured the information in the reference answer, given the context of the passage.

This dataset serves as a benchmark for training and evaluating learned metrics. By providing a wide range of model errors and correct paraphrases, MOCHA allows for the development of metrics that go beyond simple surface-level similarity.

The LERC Metric

LERC is a BERT-based learned metric that treats evaluation as a regression task. Unlike traditional metrics that only compare a candidate to a reference, LERC takes four distinct inputs: the Passage, the Question, the Reference answer, and the Candidate answer.

The LERC model architecture utilizes BERT to process the passage, question, reference, and candidate answer to predict a correctness score.

The model follows a two-stage training process:

  1. Pre-training: LERC is first pre-trained on multiple-choice QA tasks (e.g., RACE, BoolQ). This stage is crucial as it teaches the model the functional relationship between a passage and a valid answer.
  2. Fine-tuning: The model is then fine-tuned on the MOCHA dataset to predict the 1–5 human score using a regression layer attached to the BERT [CLS] token.

Key Results and Insights

LERC significantly outperforms traditional baselines. It achieves an average Pearson correlation of 0.75 with human judgments, compared to 0.48 for BLEU-1 and 0.45 for BERTScore.

A key finding is the "Passage-Agnostic Trap." Traditional metrics penalize correct answers that use different wording from the reference because they do not look at the source text. LERC avoids this by leveraging the passage to verify semantic equivalence. Furthermore, LERC demonstrates superior robustness on a "Minimal Pairs" stress test—200 pairs of candidates differing by small but critical linguistic features (like negation or name swaps). LERC correctly identified the better candidate in 80.3% of cases, while BLEU-1 performed near chance (53.5%).

Limitations and Error Analysis

Despite its success, LERC is not a perfect substitute for human judgment. Error analysis reveals that the model still struggles with certain semantic nuances and sometimes over-relies on reference-candidate similarity rather than fully utilizing the passage.

Examples of error sources where LERC scores deviate from human judgments, highlighting challenges in passage use and semantic interpretation.

Common error sources include:

  • Passage Use (22.5%): Failing to correctly verify facts against the source text.
  • Opposite Meaning (15%): Missing subtle negations or antonyms that flip the correctness of an answer.
  • Semantic Roles: Difficulty in distinguishing between similar entities performing different roles in a complex sentence.

Conclusion

MOCHA and LERC represent a significant step toward reliable evaluation for generative reading comprehension. By incorporating the passage and question into the evaluation process and utilizing a pre-trained understanding of QA, LERC provides a much more accurate reflection of human judgment than n-gram-based metrics. Future work will focus on improving the model's sensitivity to complex logic and linguistic structures.

Summary updated

BibTeX
@inproceedings{chen2020mocha,
  author = {Anthony Chen and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
  title = {{MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=3L1bhI0iKjo},
}
Figure