TL;DR
Existing Chinese Machine Reading Comprehension (MRC) benchmarks focus predominantly on short modern texts or second-language learners. This paper introduces Native Chinese Reader (NCR), a challenging multiple-choice benchmark collected from Chinese high-school exams that includes lengthy modern articles, classical prose, and poetry. While the strongest AI models achieve ~60% test accuracy, native speakers reach ~79%, exposing a substantial 20-point gap in native-level linguistic and cultural reasoning.
What Problem Does This Paper Address?
While English MRC benchmarks have evolved from simple span extraction to complex multi-step reasoning, Chinese MRC benchmarks have lagged behind native fluency demands. Prior Chinese datasets typically suffer from three constraints:
- Short context windows: Reading passages rarely exceed a few hundred characters (e.g., $C^3$ averages just 116 characters).
- Simplified, non-native tasks: Many datasets evaluate Chinese as a second language (CSL), focusing on basic factual extraction rather than deep cultural, historical, or literary inference.
- Absence of Classical Chinese: Standard Chinese education heavily tests Classical Chinese (Wenyanwen) and traditional poetry—genres with concise, highly flexible grammar, shifting parts of speech, and omitted subjects/objects.
Methods
The authors benchmark NCR using both standard and state-of-the-art MRC approaches:
- Pretrained Language Model Fine-Tuning: The authors evaluate standard BERT architectures, ERNIE, Chinese RoBERTa-large, and MacBERT variants.
- Data Augmentation: To test training scalability, the authors collect an auxiliary dataset of 6,000 documents and 13,000 questions from primary-school Chinese examinations.
- Open Competition Baseline: A 3-month public competition (141 participating teams) yielded an industry-developed hierarchical system combining Okapi BM25 document retrieval with a custom XLNet-based model pretrained on proprietary corpora.
- Human Benchmark: Evaluated on 30 native-speaking college sophomores (computer science majors who had not taken Chinese language courses for two years), testing 120 questions with $\ge 3$ annotations per item.
Datasets
- Native Chinese Reader (NCR): Contains 8,390 documents and 20,477 questions sourced from open-access high-school Chinese exams, split document-wise into train (6,315), validation (1,000), and test (1,073).
- Novel Held-Out Test Set: Includes 73 unseen documents with 193 original questions written by high-school teachers to prevent data contamination from online exam solutions.
- Scale & Diversity: Passages average 1,024 characters (up to 4,640 characters). Roughly 25% of passages are Classical Chinese prose or poetry.
- Negation Reasoning: Over 56% of questions ask the reader to identify an incorrect option (e.g., contains phrases like “不正确” or “不符合”).
- Comparison Datasets: Evaluated alongside prior Chinese MRC benchmarks, primarily $C^3$ ($C^3\text{-M}$ and $C^3\text{-D}$), CMRC 2017–2019, ChID, DRCD, and DuReader.
Key Insights and Ideas
- AI and Humans Have Inverted Strengths:
- AI models perform worst on classical poetry (best model: 41.8%), whereas human readers excel most on poetry (91.7%).
- Conversely, the performance gap between AI and humans is narrowest on sentiment analysis questions (human 63.3% vs. model 56.4%) and widest on semantic/word-level reasoning in classical contexts (human 90.5% vs. model 58.3%).
- Modern Pretraining Fails to Zero-Shot Classical Chinese: Filtering out Classical Chinese from training drops performance on classical texts from 38.7% to 29.7% while modern text performance remains flat. General Chinese language models do not naturally transfer modern semantics to classical syntax.
- Retrieval Bottlenecks Global Reasoning: The competition-winning retrieval model (BM25 + XLNet) struggled significantly with short texts and sentiment analysis because local retrieval misses subtle document-wide thematic context.
Key Results
- Human Baseline: 79.17% overall test accuracy (inter-annotator agreement: 96.7% of questions had agreement between at least two annotators).
- Baseline Fine-Tuned Models:
- Standard BERT-Chinese: 39.46%
- MacBERT-large: 47.80%
- MacBERT-large (with primary-school data augmentation): 50.21%
- Competition Winner (BM25 + XLNet): 59.85%, beating the author-trained baselines by nearly 10% but still falling ~20% shy of human comprehension.
Limitations and Future Work
- Classical Pretraining Deficit: Modern Chinese pretrained representations lack classical vocabulary and grammatical nuances, indicating a need for dedicated historical/classical corpora during pretraining.
- Long-Context Modeling: Standard transformer window limits (typically 512 tokens) force models to either truncate or rely on sparse retrieval methods like BM25, which drop crucial global context necessary for literary analysis.
- Proprietary Pretraining Data: The winning competition model utilized private industry corpora that could not be openly redistributed, underscoring the need for open-source foundation models capable of native-level literary reasoning.
Summary updated