CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Lei Li, Xiang Chen, Shumin Deng, Luoqiu Li, Hongbin Ye, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Mosha Chen, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Huajun Chen, Buzhou Tang, Qingcai Chen

Abstract

Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.

Introduction

While biomedical Natural Language Processing (BioNLP) has seen massive leaps in performance recently, the progress has been overwhelmingly "Anglo-centric." Benchmarks like BLURB and datasets like PubMedQA have catalyzed English medical AI, but these successes don't easily translate to other languages.

Chinese presents unique hurdles: it lacks explicit word boundaries, features a high degree of polysemy (where meanings change based on context or tone), and often mixes formal clinical terminology with highly colloquial "patient-speak" in search queries. Before CBLUE, there was no standardized "north star" for researchers to evaluate how well their models actually understood Chinese medical text.

The CBLUE Benchmark

CBLUE (Chinese Biomedical Language Understanding Evaluation) is the first comprehensive, community-driven benchmark designed to standardize and accelerate medical AI research for the Chinese language. The benchmark was constructed by aggregating data from clinical trials, Electronic Health Records (EHRs), medical textbooks, and real-world search engine logs (from Alibaba’s KUAKE).

Overview of CBLUE datasets, tasks, and data splits.

Key methodological highlights include:

  • Task Diversity: The benchmark covers token-level (NER), sequence-level (classification), and sequence-pair (semantic similarity) tasks.
  • Rigorous Annotation: Each sample was annotated by 3 to 5 domain experts (doctors from Class A hospitals), achieving a Fleiss’ Kappa score of 0.9.
  • Utility-Preserving Anonymization: To handle sensitive EHR data, the authors used specialized techniques to strip personal identifiers while maintaining medical utility.
  • Systematic Baselines: The authors evaluated 11 different Chinese pre-trained language models (PTLMs), including BERT, RoBERTa, and the domain-specific PCL-MedBERT.

Experimental Analysis and Key Insights

The most striking finding is the performance gap between machines and humans. While state-of-the-art models are catching up to humans in general-domain Chinese, they lag by a wide margin in the medical domain. The best model averaged a score of 70.0, while the human majority vote reached 77.1.

Error Analysis

Detailed error analysis reveals the specific linguistic challenges models face. In the CMeEE (NER) task, ambiguity and entity overlap are major hurdles. In the KUAKE-QIC task, which involves real-world search queries, "multiple triggers" and "colloquialism" are the primary sources of error.

Error analysis for CMeEE (NER) and KUAKE-QIC (Intent Classification).

Key Findings

  • Domain-Specific Models Aren't a Silver Bullet: Surprisingly, PCL-MedBERT (trained specifically on medical corpora) did not consistently outperform general-purpose models like RoBERTa-large. This suggests that current medical pre-training might not be capturing the full nuance of the Chinese language.
  • The Colloquialism Trap: Models struggled significantly with the KUAKE datasets. These search queries are often ungrammatical or use non-technical symptom descriptions. Bridging the gap between "doctor-speak" and "patient-speak" remains a major hurdle.
  • Long-Tail Challenges: Medical data follows a Zipfian distribution. Models performed poorly on rare diseases and entities, which is a critical failure point for real-world clinical safety.

Conclusion and Future Work

CBLUE 1.0 represents a significant step toward standardizing Chinese medical AI evaluation. However, the results show that current models still trail significantly behind human experts. Future work will focus on expanding the benchmark to include more dynamic tasks, such as medical dialogue generation and multi-modal tasks that combine text with medical imaging. The team has released the code and a leaderboard on the Tianchi platform to invite global collaboration in closing the gap between machine and human understanding in healthcare.

Summary updated

Abstract

Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at \url{https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us}.
BibTeX
@inproceedings{zhang2022cblue,
  author = {Ningyu Zhang and Zhen Bi and Xiaozhuan Liang and Lei Li and Xiang Chen and Shumin Deng and Luoqiu Li and Hongbin Ye and Xin Shang and Kangping Yin and Chuanqi Tan and Jian Xu and Mosha Chen and Fei Huang and Luo Si and Yuan Ni and Guotong Xie and Zhifang Sui and Baobao Chang and Hui Zong and Zheng Yuan and Linfeng Li and Jun Yan and Hongying Zan and Kunli Zhang and Huajun Chen and Buzhou Tang and Qingcai Chen},
  title = {{CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.544},
}
Figure