MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Abstract

Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers. However, progress is impeded by existing generation metrics, which rely on token overlap and are agnostic to the nuances of reading comprehension. To address this, we introduce a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. MOCHA contains 40K human judgement scores on model outputs from 6 diverse question answering datasets and an additional set of minimal pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy, outperforming baselines by 14 to 26 absolute percentage points while leaving significant room for improvement. MOCHA presents a challenging problem for developing accurate and robust generative reading comprehension metrics.

Introduction

Generative reading comprehension (RC) offers significant flexibility by allowing models to produce free-form answers to open-ended questions. However, evaluating these models remains a major challenge. Standard metrics like BLEU, ROUGE, and METEOR rely primarily on token overlap and are task-agnostic, meaning they do not consider the context of the source passage. Consequently, a model might receive a low score for a perfect paraphrase or a high score for a factually incorrect answer that happens to share many words with the reference. To bridge this gap, we introduce MOCHA and LERC.

The MOCHA Dataset

MOdeling Correctness with Human Annotations (MOCHA) is a comprehensive benchmark designed to train and evaluate generative RC metrics. It consists of 40,000 human judgment scores (on a 1-5 scale) collected for model outputs across six diverse QA datasets, including NarrativeQA, CosmosQA, and DROP.

Examples of MOCHA instances showing human scores for various candidate answers.

Beyond standard annotations, MOCHA includes a set of 200 "minimal pairs"—manually crafted answer pairs where one is slightly better than the other due to a specific linguistic feature like negation or coreference. This serves as a rigorous stress test for metric robustness, revealing that n-gram metrics often perform no better than chance when faced with such nuances.

LERC: A Learned Metric for Reading Comprehension

Using MOCHA, the authors developed LERC (Learned Evaluation metric for Reading Comprehension). LERC is a BERT-based model designed to mimic human judgment by considering the full context of the task.

LERC architecture: The model processes the passage, question, reference, and candidate answer to predict a correctness score.

LERC's architecture processes a four-way tuple: the Passage, the Question, the Reference Answer, and the Candidate Answer. This allows the metric to determine if a candidate answer is supported by the text, regardless of its lexical similarity to the reference. The model is first pre-trained on multiple-choice QA datasets to learn the characteristics of correct answers and then fine-tuned on MOCHA to predict human scores.

Key Results and Insights

LERC significantly outperforms traditional metrics across several dimensions:

  • Correlation with Humans: LERC achieves a 10 to 36 absolute Pearson point improvement over baselines like BLEU and BERTScore on held-out annotations.
  • Robustness on Minimal Pairs: On the difficult minimal pairs test, LERC achieved 80.3% accuracy in picking the better answer, while the next best baseline (BERT STS-B) trailed at 66.6%, and BLEU-1 was barely better than a coin flip at 53.5%.
  • Generalization: The metric demonstrates strong performance even on "out-of-dataset" tasks, indicating it learns general principles of RC correctness rather than dataset-specific patterns.

Conclusion

MOCHA and LERC represent a significant step toward more meaningful evaluation of generative RC models. By incorporating the source passage and question into the evaluation process, LERC moves beyond simple string matching toward a more semantic understanding of correctness. While LERC shows substantial improvements, a 20% gap remains on minimal pairs, particularly involving complex semantic roles and large lexical gaps. Future work will focus on refining these metrics to handle finer linguistic nuances and further closing the gap with human judgment.

Summary updated

Abstract

Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers. However, progress is impeded by existing generation metrics, which rely on token overlap and are agnostic to the nuances of reading comprehension. To address this, we introduce a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. MOCHA contains 40K human judgement scores on model outputs from 6 diverse question answering datasets and an additional set of minimal pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy, outperforming baselines by 14 to 26 absolute percentage points while leaving significant room for improvement. MOCHA presents a challenging problem for developing accurate and robust generative reading comprehension metrics.
BibTeX
@inproceedings{chen2020mocha,
  author = {Anthony Chen and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
  title = {{MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=3L1bhI0iKjo},
}

Loading related content...

Figure