Abstract
Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
Introduction
While biomedical Natural Language Processing (BioNLP) has seen massive leaps in performance recently, the progress has been overwhelmingly "Anglo-centric." Benchmarks like BLURB and datasets like PubMedQA have catalyzed English medical AI, but these successes don't easily translate to other languages.
Chinese presents unique hurdles: it lacks explicit word boundaries, features a high degree of polysemy (where meanings change based on context or tone), and often mixes formal clinical terminology with highly colloquial "patient-speak" in search queries. Before CBLUE, there was no standardized "north star" for researchers to evaluate how well their models actually understood Chinese medical text.
The CBLUE Benchmark
CBLUE (Chinese Biomedical Language Understanding Evaluation) is the first comprehensive, community-driven benchmark designed to standardize and accelerate medical AI research for the Chinese language. The benchmark was constructed by aggregating data from clinical trials, Electronic Health Records (EHRs), medical textbooks, and real-world search engine logs (from Alibaba’s KUAKE).
Key methodological highlights include:
- Task Diversity: The benchmark covers token-level (NER), sequence-level (classification), and sequence-pair (semantic similarity) tasks.
- Rigorous Annotation: Each sample was annotated by 3 to 5 domain experts (doctors from Class A hospitals), achieving a Fleiss’ Kappa score of 0.9.
- Utility-Preserving Anonymization: To handle sensitive EHR data, the authors used specialized techniques to strip personal identifiers while maintaining medical utility.
- Systematic Baselines: The authors evaluated 11 different Chinese pre-trained language models (PTLMs), including BERT, RoBERTa, and the domain-specific PCL-MedBERT.
Experimental Analysis and Key Insights
The most striking finding is the performance gap between machines and humans. While state-of-the-art models are catching up to humans in general-domain Chinese, they lag by a wide margin in the medical domain. The best model averaged a score of 70.0, while the human majority vote reached 77.1.
Error Analysis
Detailed error analysis reveals the specific linguistic challenges models face. In the CMeEE (NER) task, ambiguity and entity overlap are major hurdles. In the KUAKE-QIC task, which involves real-world search queries, "multiple triggers" and "colloquialism" are the primary sources of error.
Key Findings
- Domain-Specific Models Aren't a Silver Bullet: Surprisingly, PCL-MedBERT (trained specifically on medical corpora) did not consistently outperform general-purpose models like RoBERTa-large. This suggests that current medical pre-training might not be capturing the full nuance of the Chinese language.
- The Colloquialism Trap: Models struggled significantly with the KUAKE datasets. These search queries are often ungrammatical or use non-technical symptom descriptions. Bridging the gap between "doctor-speak" and "patient-speak" remains a major hurdle.
- Long-Tail Challenges: Medical data follows a Zipfian distribution. Models performed poorly on rare diseases and entities, which is a critical failure point for real-world clinical safety.
Conclusion and Future Work
CBLUE 1.0 represents a significant step toward standardizing Chinese medical AI evaluation. However, the results show that current models still trail significantly behind human experts. Future work will focus on expanding the benchmark to include more dynamic tasks, such as medical dialogue generation and multi-modal tasks that combine text with medical imaging. The team has released the code and a leaderboard on the Tianchi platform to invite global collaboration in closing the gap between machine and human understanding in healthcare.
Summary updated