Overview of the PromptCBLUE Shared Task in CHIP2023

Wei Zhu, Xiaoling Wang, Mosha Chen, Buzhou TangarXiv 2023 · DOI · arXiv

Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing, yet their application in specialized fields like medicine remains challenging. While proprietary models like ChatGPT demonstrate impressive medical knowledge, privacy concerns and the need for local deployment have spurred the development of open-source Chinese LLMs. However, traditional benchmarks like CBLUE are not optimized for the instruction-following nature of modern LLMs. To address this, the PromptCBLUE shared task was introduced at CHIP2023, establishing a comprehensive benchmark to evaluate Chinese LLMs across 18 specialized medical tasks.

Methodology: Reformulating Medical NLP

The core innovation of this work is the creation of PromptCBLUE, a large-scale multi-task dataset where traditional medical NLP problems are reformulated into natural language prompts. This shift allows for a direct evaluation of how models handle complex medical instructions.

Overview of the PromptCBLUE framework and task categories.

The shared task was structured into two distinct tracks to compare different adaptation strategies:

  1. Parameter-Efficient Fine-Tuning (PEFT) Track: Participants fine-tuned open-source backbones (e.g., ChatGLM2-6B, Baichuan-13B) by updating a minimal fraction of parameters, primarily using Low-Rank Adaptation (LoRA).
  2. In-Context Learning (ICL) Track: Models were evaluated in a zero-shot or few-shot setting without parameter updates. Success depended on optimizing the "context" through sophisticated demonstration selection.

Dataset Composition

PromptCBLUE is a "super-dataset" comprising 18 sub-tasks categorized into five cohorts: Medical Information Extraction, Medical Text Classification, Medical Natural Language Inference (NLI), Symptom Status Understanding, and Medical Content Generation.

Statistical breakdown of the 18 sub-tasks within the PromptCBLUE benchmark.

As shown in the table above, the benchmark includes approximately 87,100 training pairs. The tasks vary significantly in complexity, with prompt lengths ranging from roughly 60 to over 800 tokens, reflecting the diverse requirements of clinical documentation and dialogue.

Key Insights and Results

The competition attracted over 600 teams, providing a robust snapshot of the current state of Chinese medical LLMs. Several critical findings emerged:

  • The Fine-Tuning Gap: A significant performance chasm exists between PEFT and ICL. The top PEFT score reached 71.38, whereas the best ICL score was only 40.27. This indicates that for high-stakes clinical applications, prompt engineering alone is currently insufficient; targeted fine-tuning remains essential.
  • Model Scaling: Larger models (13B parameters) consistently outperformed smaller counterparts (7B). The Baichuan-13B-Chat model proved to be a particularly effective backbone for Chinese medical tasks.
  • Optimization Strategies: In the ICL track, the most successful teams treated demonstration selection as a "Knapsack Problem," balancing semantic relevance against token limits. In the PEFT track, data augmentation and robust training techniques like random token masking were key differentiators.

Conclusion and Future Directions

PromptCBLUE represents a significant step toward standardized evaluation of medical LLMs. While open-source models show great potential, their performance in zero-shot or few-shot medical reasoning still lags behind proprietary models. Future research will focus on closing this ICL gap, refining evaluation metrics to better reflect clinical utility, and exploring the capabilities of even larger open-source models (30B-70B+) as they become more accessible.

Summary updated

Abstract

This paper presents an overview of the PromptCBLUE shared task (http://cips-chip.org.cn/2023/eval1) held in the CHIP-2023 Conference. This shared task reformualtes the CBLUE benchmark, and provide a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. Two different tracks are held: (a) prompt tuning track, investigating the multitask prompt tuning of LLMs, (b) probing the in-context learning capabilities of open-sourced LLMs. Many teams from both the industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.
BibTeX
@misc{zhu2023overview,
  author = {Wei Zhu and Xiaoling Wang and Mosha Chen and Buzhou Tang},
  title = {{Overview of the PromptCBLUE Shared Task in CHIP2023}},
  year = {2023},
  eprint = {2312.17522},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.17522},
}
Figure