A Sentence Cloze Dataset for Chinese Machine Reading Comprehension

Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, Guoping Hu

Abstract

Owing to the continuous efforts by the Chinese NLP community, more and more Chinese machine reading comprehension datasets become available. To add diversity in this area, we propose a new task called Sentence Cloze-style Machine Reading Comprehension (SC-MRC). The proposed task aims to fill the right candidate sentence into the passage that has several blanks. We built a Chinese dataset called CMRC 2019 to evaluate the difficulty of the SC-MRC task. Moreover, to add more difficulties, we also made fake candidates that are similar to the correct ones, which requires the machine to judge their correctness in the context. The proposed dataset contains over 100K blanks (questions) within over 10K passages, which was originated from Chinese narrative stories. To evaluate the dataset, we implement several baseline systems based on the pre-trained models, and the results show that the state-of-the-art model still underperforms human performance by a large margin.

Introduction

Most Machine Reading Comprehension (MRC) benchmarks focus on "span extraction" or "word cloze" tasks. While models have become proficient at these, critics argue they often perform sophisticated pattern matching rather than true narrative understanding. In Chinese NLP, there has been a lack of sentence-level inference tasks. To address this, we introduce the Sentence Cloze-style MRC (SC-MRC) task and the CMRC 2019 dataset. This task requires models to understand the logical flow and coherence between sentences to restore a passage with missing segments.

An example of the SC-MRC task from the CMRC 2019 dataset, featuring multiple blanks within a narrative story.

The CMRC 2019 Dataset

CMRC 2019 is a large-scale benchmark specifically designed for sentence-level inference. It consists of over 100,000 blanks across more than 10,000 passages sourced from Chinese children's narrative stories. These stories are ideal for testing logical progression and narrative flow.

A unique feature of CMRC 2019 is the inclusion of fake candidates. These are sentences taken from the same story but outside the specific passage used. Because they share the same topic and named entities, models cannot rely on simple vocabulary overlap; they must comprehend the specific context to distinguish the correct sentence from the distractor. This design successfully neutralizes "entity matching" shortcuts.

Methodology and Baselines

We evaluated the dataset using several state-of-the-art pre-trained models, including BERT (and its Whole Word Masking variant, BERT-wwm) and RoBERTa. The models concatenate a candidate sentence with the passage using a [SEP] token and predict the probability of that sentence fitting into each available blank.

Performance is measured using two primary metrics:

  1. Question-level Accuracy (QAC): The percentage of individual blanks filled correctly.
  2. Passage-level Accuracy (PAC): The percentage of passages where every single blank is filled correctly. PAC is a significantly harder metric as it requires global consistency across the narrative.

Results and Discussion

The experimental results reveal a substantial "reasoning gap" between humans and current AI models. While the strongest model, RoBERTa-wwm-ext-large, achieves a respectable 81.7% QAC on the test set, its PAC drops to 23.0%. In contrast, humans achieve a 75.0% PAC, outperforming the best model by 52 percentage points.

Performance comparison showing the significant gap between human performance and various baseline models, particularly in passage-level accuracy (PAC).

These findings suggest that while pre-trained models are adept at local matching, they struggle with the global coherence required to reconstruct a full narrative. They often make locally plausible choices that break the global logic of the story.

Conclusion

CMRC 2019 shifts the focus of Chinese MRC from token-matching to sentence-level reasoning. The results demonstrate that current state-of-the-art models still lack the ability to perform global narrative optimization. We hope this dataset encourages the community to move beyond simple pattern matching toward deeper narrative comprehension.

Summary updated

Abstract

Owing to the continuous efforts by the Chinese NLP community, more and more Chinese machine reading comprehension datasets become available. To add diversity in this area, in this paper, we propose a new task called Sentence Cloze-style Machine Reading Comprehension (SC-MRC). The proposed task aims to fill the right candidate sentence into the passage that has several blanks. We built a Chinese dataset called CMRC 2019 to evaluate the difficulty of the SC-MRC task. Moreover, to add more difficulties, we also made fake candidates that are similar to the correct ones, which requires the machine to judge their correctness in the context. The proposed dataset contains over 100K blanks (questions) within over 10K passages, which was originated from Chinese narrative stories. To evaluate the dataset, we implement several baseline systems based on the pre-trained models, and the results show that the state-of-the-art model still underperforms human performance by a large margin. We release the dataset and baseline system to further facilitate our community. Resources available through https://github.com/ymcui/cmrc2019
BibTeX
@inproceedings{cui2020sentence,
  author = {Yiming Cui and Ting Liu and Ziqing Yang and Zhipeng Chen and Wentao Ma and Wanxiang Che and Shijin Wang and Guoping Hu},
  title = {{A Sentence Cloze Dataset for Chinese Machine Reading Comprehension}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=snG72wqFsX},
}
Figure