Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

Shusheng Xu, Yichen Liu, Xiaoyu Yi, Siyuan Zhou, Huizi Li, Yi Wu

TL;DR

Existing Chinese Machine Reading Comprehension (MRC) benchmarks focus predominantly on short modern texts or second-language learners. This paper introduces Native Chinese Reader (NCR), a challenging multiple-choice benchmark collected from Chinese high-school exams that includes lengthy modern articles, classical prose, and poetry. While the strongest AI models achieve ~60% test accuracy, native speakers reach ~79%, exposing a substantial 20-point gap in native-level linguistic and cultural reasoning.

What Problem Does This Paper Address?

While English MRC benchmarks have evolved from simple span extraction to complex multi-step reasoning, Chinese MRC benchmarks have lagged behind native fluency demands. Prior Chinese datasets typically suffer from three constraints:

  1. Short context windows: Reading passages rarely exceed a few hundred characters (e.g., $C^3$ averages just 116 characters).
  2. Simplified, non-native tasks: Many datasets evaluate Chinese as a second language (CSL), focusing on basic factual extraction rather than deep cultural, historical, or literary inference.
  3. Absence of Classical Chinese: Standard Chinese education heavily tests Classical Chinese (Wenyanwen) and traditional poetry—genres with concise, highly flexible grammar, shifting parts of speech, and omitted subjects/objects.

Methods

The authors benchmark NCR using both standard and state-of-the-art MRC approaches:

  • Pretrained Language Model Fine-Tuning: The authors evaluate standard BERT architectures, ERNIE, Chinese RoBERTa-large, and MacBERT variants.
  • Data Augmentation: To test training scalability, the authors collect an auxiliary dataset of 6,000 documents and 13,000 questions from primary-school Chinese examinations.
  • Open Competition Baseline: A 3-month public competition (141 participating teams) yielded an industry-developed hierarchical system combining Okapi BM25 document retrieval with a custom XLNet-based model pretrained on proprietary corpora.
  • Human Benchmark: Evaluated on 30 native-speaking college sophomores (computer science majors who had not taken Chinese language courses for two years), testing 120 questions with $\ge 3$ annotations per item.

Datasets

  • Native Chinese Reader (NCR): Contains 8,390 documents and 20,477 questions sourced from open-access high-school Chinese exams, split document-wise into train (6,315), validation (1,000), and test (1,073).
  • Novel Held-Out Test Set: Includes 73 unseen documents with 193 original questions written by high-school teachers to prevent data contamination from online exam solutions.
  • Scale & Diversity: Passages average 1,024 characters (up to 4,640 characters). Roughly 25% of passages are Classical Chinese prose or poetry.
  • Negation Reasoning: Over 56% of questions ask the reader to identify an incorrect option (e.g., contains phrases like “不正确” or “不符合”).
  • Comparison Datasets: Evaluated alongside prior Chinese MRC benchmarks, primarily $C^3$ ($C^3\text{-M}$ and $C^3\text{-D}$), CMRC 2017–2019, ChID, DRCD, and DuReader.

Key Insights and Ideas

  • AI and Humans Have Inverted Strengths:
  • AI models perform worst on classical poetry (best model: 41.8%), whereas human readers excel most on poetry (91.7%).
  • Conversely, the performance gap between AI and humans is narrowest on sentiment analysis questions (human 63.3% vs. model 56.4%) and widest on semantic/word-level reasoning in classical contexts (human 90.5% vs. model 58.3%).
  • Modern Pretraining Fails to Zero-Shot Classical Chinese: Filtering out Classical Chinese from training drops performance on classical texts from 38.7% to 29.7% while modern text performance remains flat. General Chinese language models do not naturally transfer modern semantics to classical syntax.
  • Retrieval Bottlenecks Global Reasoning: The competition-winning retrieval model (BM25 + XLNet) struggled significantly with short texts and sentiment analysis because local retrieval misses subtle document-wide thematic context.

Key Results

  • Human Baseline: 79.17% overall test accuracy (inter-annotator agreement: 96.7% of questions had agreement between at least two annotators).
  • Baseline Fine-Tuned Models:
  • Standard BERT-Chinese: 39.46%
  • MacBERT-large: 47.80%
  • MacBERT-large (with primary-school data augmentation): 50.21%
  • Competition Winner (BM25 + XLNet): 59.85%, beating the author-trained baselines by nearly 10% but still falling ~20% shy of human comprehension.

Limitations and Future Work

  • Classical Pretraining Deficit: Modern Chinese pretrained representations lack classical vocabulary and grammatical nuances, indicating a need for dedicated historical/classical corpora during pretraining.
  • Long-Context Modeling: Standard transformer window limits (typically 512 tokens) force models to either truncate or rely on sparse retrieval methods like BM25, which drop crucial global context necessary for literary analysis.
  • Proprietary Pretraining Data: The winning competition model utilized private industry corpora that could not be openly redistributed, underscoring the need for open-source foundation models capable of native-level literary reasoning.

Summary updated

Abstract

We present Native Chinese Reader (NCR), a new machine reading comprehension MRC) dataset with particularly long articles in both modern and classical Chinese. NCR is collected from the exam questions for the Chinese course in China’s high schools, which are designed to evaluate the language proficiency of native Chinese youth. Existing Chinese MRC datasets are either domain-specific or focusing on short contexts of a few hundred characters in modern Chinese only. By contrast, NCR contains 8390 documents with an average length of 1024 characters covering a wide range of Chinese writing styles, including modern articles, classical literature and classical poetry. A total of 20477 questions on these documents also require strong reasoning abilities and common sense to figure out the correct answers. We implemented multiple baseline models using popular Chinese pre-trained models and additionally launched an online competition using our dataset to examine the limit of current methods. The best model achieves 59% test accuracy while human evaluation shows an average accuracy of 79%, which indicates a significant performance gap between current MRC models and native Chinese speakers.
BibTeX
@inproceedings{xu2021native,
  author = {Shusheng Xu and Yichen Liu and Xiaoyu Yi and Siyuan Zhou and Huizi Li and Yi Wu},
  title = {{Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=GEcWUTN1v1v},
}

Loading related content...

Figure