CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Mosha Chen, Bi Zhen, Xiaozhuan Liang, Lei Li, Ningyu Zhang, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, Qingcai Chen

TL;DR

This paper introduces CBLUE, the first comprehensive benchmark for evaluating Chinese biomedical language understanding. It addresses the critical gap of English-centric NLP resources by providing a diverse set of tasks and a platform to foster research in Chinese biomedical NLP, highlighting that current state-of-the-art models still significantly lag behind human performance.

What Problem Does This Paper Address?

The field of Artificial Intelligence (AI) is making significant strides in biomedical language understanding, leading to promising applications in healthcare. However, the vast majority of existing benchmarks and research efforts are focused on English. This creates a substantial barrier for developing and evaluating AI models for other languages, particularly for Chinese, which has a quarter of the world's population and unique linguistic characteristics. This English-centricity limits the global applicability and advancement of biomedical NLP. CBLUE aims to bridge this gap by providing a much-needed evaluation framework for Chinese biomedical language understanding.

Methods

CBLUE is a benchmark comprising eight diverse natural language understanding tasks tailored for the biomedical domain in Chinese. These tasks include:

  • Named Entity Recognition (NER): Identifying and classifying entities like diseases, symptoms, drugs, etc. (CMeEE dataset).
  • Information Extraction (IE): Identifying entities and their relationships (CMeIE dataset).
  • Clinical Diagnosis Normalization: Standardizing diagnostic terms to ICD-10 (CHIP-CDN dataset).
  • Single-Sentence/Sentence-Pair Classification:
    • Clinical Trial Classification (CHIP-CTC dataset).
  • Semantic Textual Similarity (CHIP-STS dataset), with a unique non-i.i.d. setting for transfer learning across different disease types.
  • Query Intent Classification (KUAKE-QIC dataset).
  • Query-Title Relevance (KUAKE-QTR dataset).
  • Query-Query Relevance (KUAKE-QQR dataset).

The paper also details a rigorous data collection process involving clinical trials, EHRs, medical forums, textbooks, and search engine logs. All data is anonymized and reviewed by an IRB committee. Annotation is performed by multiple crowd workers with a majority vote system, and inter-rater agreement is reported to be high.

Datasets

CBLUE is built upon several datasets, some of which were introduced or adapted for this benchmark:

  • CMeEE: Chinese Medical Named Entity Recognition.
  • CMeIE: Chinese Medical Information Extraction.
  • CHIP-CDN: CHIP Clinical Diagnosis Normalization.
  • CHIP-CTC: CHIP Clinical Trial Classification.
  • CHIP-STS: CHIP Semantic Textual Similarity.
  • KUAKE-QIC: KUAKE Query Intent Classification.
  • KUAKE-QTR: KUAKE Query Title Relevance.
  • KUAKE-QQR: KUAKE Query-Query Relevance.

These datasets are designed to cover a broad spectrum of biomedical NLP tasks, with a focus on real-world distribution and unique Chinese linguistic challenges.

Key Insights and Ideas

  • The "Chinese Wall" in Biomedical NLP: The paper strongly highlights the significant gap in biomedical NLP research due to the dominance of English-centric resources. CBLUE is a crucial step towards democratizing this field for Chinese speakers.
  • Human Performance as the Ceiling: Even with advanced pre-trained models, performance on CBLUE tasks is significantly lower than human performance, indicating substantial room for improvement and the complexity of Chinese biomedical language.
  • Linguistic Nuances Matter: The paper points out that Chinese, being a tonal and often colloquial language, presents unique challenges for NLP models, even in specialized domains like medicine.
  • Model Size Isn't Everything: The results suggest that larger pre-trained models don't always guarantee better performance across all tasks. Smaller models like ALBERT-tiny can achieve comparable results on certain tasks, hinting at efficiency opportunities.
  • Whole Word Masking's Mixed Results: While beneficial in some cases, whole word masking doesn't consistently lead to better performance, suggesting task-specific adaptations are still crucial.

Key Results

The paper reports empirical results using 11 pre-trained Chinese language models on the CBLUE benchmark. Key findings include:

  • State-of-the-art models lag behind humans: Across most tasks, the best-performing models achieve scores significantly lower than the human ceiling. For example, on the CMeEE task, the best model achieved an F1 score of 62.4, while human performance was 67.0.
  • Model performance varies by task: Some tasks, like QIC, QTR, and QQR, proved particularly challenging for current models, with performance gaps widening compared to human baselines.
  • PCL-MedBERT's performance: While PCL-MedBERT, a domain-specific model, showed promise in specific cases (e.g., identifying medical terminologies), it did not consistently outperform general pre-trained models on all CBLUE tasks, underscoring the benchmark's difficulty.
  • High inter-rater agreement: The annotation process yielded high inter-rater agreement (κ = 0.9), indicating the reliability of the created datasets.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Task Coverage: CBLUE, while comprehensive, does not cover all potential biomedical NLP tasks, such as medical dialogue generation or diagnosis.
  • Static Benchmark: The current benchmark is static, meaning models might overfit to specific dataset characteristics. Future work could involve creating a more dynamic platform for continuous evaluation and dataset expansion.

The authors encourage researchers to contribute new datasets and tasks to expand the benchmark. They envision CBLUE as a community-driven project that will foster the development of more robust and informative Chinese biomedical language understanding models, ultimately benefiting healthcare and research.

Summary updated

Abstract

Artificial intelligence (AI), along with the recent progress in biomedical language understanding, is gradually offering great promise for medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most such benchmarks are limited to English, which has made it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us.
BibTeX
@inproceedings{chen2022cblue,
  author = {Mosha Chen and Bi Zhen and Xiaozhuan Liang and Lei Li and Ningyu Zhang and Xin Shang and Kangping Yin and Chuanqi Tan and Jian Xu and Fei Huang and Luo Si and Yuan Ni and Guotong Xie and Zhifang Sui and Baobao Chang and Hui Zong and Zheng Yuan and Linfeng Li and Jun Yan and Hongying Zan and Kunli Zhang and Buzhou Tang and Qingcai Chen},
  title = {{CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.544},
}
Figure