CLUE: A Chinese Language Understanding Evaluation Benchmark

Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, Zhenzhong Lan

TL;DR

This paper introduces CLUE, the first comprehensive benchmark for evaluating Chinese Natural Language Understanding (NLU) models. It addresses the lack of standardized evaluation for Chinese NLU, which has lagged behind English due to linguistic differences and limited datasets, and provides a large pre-training corpus and a diagnostic dataset to foster research.

What Problem Does This Paper Address?

The field of Natural Language Understanding (NLU) has seen tremendous progress, largely driven by the development of standardized evaluation benchmarks like GLUE and SuperGLUE for English. These benchmarks allow researchers to compare models across a diverse set of tasks, accelerating research and development. However, a significant gap exists for other languages, particularly Chinese. Despite Chinese being spoken by a quarter of the world's population and possessing unique linguistic characteristics, there hasn't been a comparable, comprehensive evaluation framework. This lack of standardization makes it difficult to assess and improve Chinese NLU models, hindering progress in the field and limiting the applicability of NLP advancements to a vast user base.

Methods

The CLUE benchmark is built upon several key components:

  • Task Selection: The authors curated nine diverse NLU tasks, spanning single-sentence classification, sentence-pair classification, and machine reading comprehension. These tasks were chosen based on criteria like diversity in text size and understanding required, well-definedness, moderate difficulty (challenging but solvable), representativeness of real-world applications, and tailoring to Chinese-specific linguistic phenomena.
  • Pre-training Dataset: To address the scarcity of large-scale Chinese pre-training data, the authors compiled a substantial corpus of 214 GB of raw text, totaling approximately 76 billion Chinese words. This corpus is a combination of several sources, including news, web text, Wikipedia, and e-commerce comments.
  • Diagnostic Dataset: A hand-crafted diagnostic dataset was created to specifically probe models' understanding of nuanced linguistic phenomena and logical reasoning in Chinese. This dataset is not a translation of existing English diagnostic sets but is designed to capture unique aspects of the Chinese language.
  • Evaluation Framework: CLUE provides a unified evaluation pipeline, a ranking system, and an online leaderboard with an auto-evaluation system to ensure reproducibility and facilitate easy comparison of submitted models. A toolkit, PyCLUE, is also provided for ease of use.

Datasets

  • Pre-training Datasets:
    • CLUECorpus2020-small: 14 GB of Chinese text, including news, web text, Wikipedia, and comments.
  • CLUECorpus2020: 100 GB of Chinese raw corpus retrieved from Common Crawl.
  • CLUEOSCAR: An additional 250 GB multilingual corpus, filtered to yield 100 GB of Chinese text.
  • Evaluation Tasks (9 in total):
    • Single Sentence Tasks: TNEWS (news title classification), IFLYTEK (app description classification), CLUEWSC2020 (coreference resolution).
  • Sentence Pair Tasks: AFQMC (semantic similarity), CSL (keyword recognition), OCNLI (natural language inference).
  • Machine Reading Comprehension Tasks: CMRC 2018 (answer span extraction), ChID (idiom cloze test), C³ (multiple-choice, free-form).
  • Diagnostic Dataset: A hand-crafted dataset of 514 sentence pairs covering 9 linguistic phenomena, designed specifically for Chinese.

Key Insights and Ideas

  • The Need for Language-Specific Benchmarks: The paper strongly emphasizes that simply translating English benchmarks is insufficient for languages like Chinese, which have distinct linguistic structures and phenomena. CLUE's design, especially the diagnostic dataset, highlights this by focusing on Chinese-specific linguistic challenges.
  • Model Size and Data Matter: The experiments reveal that larger models and those trained on more extensive pre-training data generally perform better across the CLUE benchmark. This aligns with general trends in NLP but is empirically demonstrated for Chinese NLU.
  • Whole Word Masking Benefits Chinese NLU: The study indicates that techniques like whole word masking (used in BERT-wwm-ext) provide a performance boost for Chinese NLU models, suggesting its importance for languages without explicit word boundaries.
  • Human Performance Gap: While state-of-the-art models show promising results, they still lag behind human performance on average by about 10%. This gap is particularly pronounced on certain tasks, indicating significant room for improvement in Chinese NLU.
  • Task Difficulty Varies: The paper highlights that tasks easy for humans might be challenging for machines (e.g., multiple-choice MRC, coreference resolution) and vice-versa. TNEWS, a single-sentence classification task, surprisingly emerged as the hardest for both humans and machines, likely due to its multi-label nature.

Key Results

  • The CLUE benchmark was evaluated using nine baseline models, including BERT, ALBERT, ERNIE, XLNet, and RoBERTa variants.
  • The best performing models, RoBERTa-wwm-ext-large and ALBERT-xxlarge, achieved an average score across all tasks that was approximately 10% lower than estimated human performance.
  • For specific tasks, models showed varying levels of success. For instance, models struggled with the diagnostic dataset, with the highest accuracy reaching only around 61%.
  • ALBERT-tiny, a significantly smaller model, showed a performance drop of up to 40% on machine reading tasks compared to larger models, underscoring the impact of model size.

Limitations and Future Work

  • Human Performance Estimates: The paper acknowledges that the human performance estimates are based on a limited number of annotators and a subset of the test data.
  • Ongoing Evolution of NLU: The authors view CLUE as an "open-ended, community-driven project." They suggest that as models reach human performance, the benchmark should be extended to newer tasks or incorporate novel evaluation forms, such as considering performance as a function of model size.
  • Further Model Development: The significant gap between model and human performance suggests a need for continued research into more robust and linguistically aware Chinese NLU models.
  • Diagnostic Dataset Expansion: The diagnostic dataset, while valuable, could be expanded to cover more linguistic phenomena and potentially incorporate more complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{xu2020clue,
  author = {Liang Xu and Hai Hu and Xuanwei Zhang and Lu Li and Chenjie Cao and Yudong Li and Yechen Xu and Kai Sun and Dian Yu and Cong Yu and Yin Tian and Qianqian Dong and Weitang Liu and Bo Shi and Yiming Cui and Junyi Li and Jun Zeng and Rongzhao Wang and Weijian Xie and Yanting Li and Yina Patterson and Zuoyu Tian and Yiwen Zhang and He Zhou and Shaoweihua Liu and Zhe Zhao and Qipeng Zhao and Cong Yue and Xinrui Zhang and Zhengliang Yang and Kyle Richardson and Zhenzhong Lan},
  title = {{CLUE: A Chinese Language Understanding Evaluation Benchmark}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.419},
}
Figure