Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

Shusheng Xu, Yichen Liu, Xiaoyu Yi, Siyuan Zhou, Huizi Li, Yi Wu

Abstract

We present Native Chinese Reader (NCR), a new machine reading comprehension (MRC) dataset with particularly long articles in both modern and classical Chinese. NCR is collected from the exam questions for the Chinese course in China's high schools, which are designed to evaluate the language proficiency of native Chinese youth. Existing Chinese MRC datasets are either domain-specific or focusing on short contexts of a few hundreds of characters in modern Chinese only. By contrast, NCR contains 8390 documents with an average length of 1024 characters covering a wide range of Chinese writing styles, including modern articles, classical literature and classical poetry. A total of 20477 questions on these documents also require strong reasoning abilities and common sense to figure out the correct answers. We implemented multiple baseline models using popular Chinese pre-trained models and additionally launched an online competition using our dataset to examine the limit of current methods. The best model achieves 59% test accuracy while human evaluation shows an average accuracy of 79%, which indicates a significant performance gap between current MRC models and native Chinese speakers. We release the dataset at https://sites.google.com/view/native-chinese-reader/.

Introduction: The Challenge of Native-Level Proficiency

Machine Reading Comprehension (MRC) has advanced significantly, yet a substantial gap remains between AI performance and native human proficiency in Chinese. Current benchmarks often fall short due to three primary limitations: short context windows (typically under 500 characters), a focus on simple extractive tasks rather than deep reasoning, and the complete absence of Classical Chinese. To achieve "native-level" proficiency, models must handle long, complex texts and understand the nuances of both modern and classical writing styles. To address these gaps, we introduce the Native Chinese Reader (NCR), a dataset derived from high school Chinese exams designed for native speakers.

The NCR Dataset

The NCR dataset represents a significant step up in complexity compared to existing benchmarks. It consists of 8,390 documents and 20,477 multiple-choice questions. Unlike previous datasets that rely on news or Wikipedia, NCR uses real-world exam materials that require sophisticated linguistic understanding and cultural context.

Comparison of NCR with existing Chinese MRC datasets. NCR stands out with significantly longer documents and the inclusion of Classical Chinese.

As shown in the comparison above, NCR's average document length is 1,024 characters, more than double that of many popular benchmarks like $C^3$ or CMRC. Furthermore, it is the only major dataset to include Classical Chinese, a literary form essential for native-level literacy.

Detailed statistics of NCR compared to other multiple-choice datasets, highlighting the substantial average and maximum document lengths.

Methodology and Benchmarking

To evaluate the difficulty of NCR, we established several benchmarks:

  • Fine-Tuning: Seven popular Chinese pre-trained models were fine-tuned, including BERT-Chinese, ERNIE, RoBERTa-large, and MacBERT.
  • Data Augmentation: To boost performance, an additional 13,000 questions from primary school Chinese exams were used to pre-train the models before fine-tuning on the high-school-level NCR.
  • The Competition Approach: An online competition revealed that the most successful approach combined an XLNet-based architecture with an Information Retrieval (IR) tool (Okapi BM25). This allowed the model to extract the most relevant chunks from long documents to fit within the model's token limit.

Key Insights: Reasoning and the Poetry Paradox

NCR moves beyond simple pattern matching. Only 0.33% of its questions are "Matching" questions, whereas nearly 48% of questions in the $C^3$ dataset can be solved this way. Instead, NCR emphasizes summary and reasoning. Over 56% of the questions in NCR require identifying the incorrect statement, forcing models to understand negations and perform logical elimination across multiple candidate options.

A particularly striking finding is the "Poetry Paradox." While humans excel at classical poetry (91.7% accuracy) due to its evocative nature, AI models perform their worst (34.6%), finding the dense grammar and abstract metaphors of classical Chinese to be a "black box."

Results and Performance Gap

The performance gap between AI and humans remains stark. While native speakers (college students) achieve an average accuracy of 79.17%, the best fine-tuned baseline (MacBERT-large with data augmentation) reached only 50.21%. Even the top-performing model from the competition, which utilized advanced IR and XLNet, only reached 59.85%. This ~20% gap underscores that native-level Chinese comprehension is still an unsolved problem for current NLP.

Conclusion and Future Directions

The NCR dataset highlights critical areas for future research. Standard Transformer models struggle with the 1,000+ character documents in NCR; while IR-based chunking helps, it can lead to a loss of global document context vital for sentiment and summary. There is a clear need for large-scale pre-training on classical corpora to bridge the linguistic divide. Furthermore, future models may need to incorporate external knowledge graphs to understand deep cultural and historical metaphors, such as specific Song Dynasty references to sadness.

Summary updated

Abstract

We present Native Chinese Reader (NCR), a new machine reading comprehension (MRC) dataset with particularly long articles in both modern and classical Chinese. NCR is collected from the exam questions for the Chinese course in China's high schools, which are designed to evaluate the language proficiency of native Chinese youth. Existing Chinese MRC datasets are either domain-specific or focusing on short contexts of a few hundreds of characters in modern Chinese only. By contrast, NCR contains 8390 documents with an average length of 1024 characters covering a wide range of Chinese writing styles, including modern articles, classical literature and classical poetry. A total of 20477 questions on these documents also require strong reasoning abilities and common sense to figure out the correct answers. We implemented multiple baseline models using popular Chinese pre-trained models and additionally launched an online competition using our dataset to examine the limit of current methods. The best model achieves 59% test accuracy while human evaluation shows an average accuracy of 79%, which indicates a significant performance gap between current MRC models and native Chinese speakers. We release the dataset at https://sites.google.com/view/native-chinese-reader/.
BibTeX
@inproceedings{xu2021native,
  author = {Shusheng Xu and Yichen Liu and Xiaoyu Yi and Siyuan Zhou and Huizi Li and Yi Wu},
  title = {{Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=GEcWUTN1v1v},
}
Figure