Abstract
Continued pretraining improved model transferability for cross-domain clinical note section classification in the presence of a small amount of in-domain labeled samples.
Introduction
In clinical informatics, section classification is a foundational task that enables downstream applications such as information extraction and clinical decision support. By identifying structural components of clinical notes—typically categorized under the Subjective, Objective, Assessment, and Plan (SOAP) framework—models can better parse complex patient records. However, a significant challenge remains: model transferability. A classifier trained at one institution often fails at another due to idiosyncratic documentation styles, varying templates, and site-specific jargon. This paper investigates the role of "continued pretraining" in bridging this performance gap.
Methodology
The authors utilized BioBERT as the base architecture and evaluated two primary unsupervised adaptation strategies:
- Domain-Adaptive Pretraining (DAPT): This involves training the model on a large, unlabeled corpus of clinical notes from the target institution using a Masked Language Modeling (MLM) objective.
- Task-Adaptive Pretraining (TAPT): A more targeted approach where the model undergoes MLM pretraining specifically on the unlabeled notes that will eventually be used for the classification task.
Furthermore, the study examined Supervised Adaptation, where a small number of labeled samples (ranging from 10 to 50) from the target domain are introduced during the fine-tuning phase. To provide a practical perspective for healthcare organizations, the researchers developed a "pseudo-inverse" function to calculate the "label value" of unsupervised pretraining—essentially quantifying how many manual annotations the pretraining process replaces.
Experimental Setup
The study's experiments were conducted across three distinct and well-regarded clinical datasets. To ensure a fair comparison, an expert physician mapped the native labels of these datasets into a unified 5-way classification scheme: Subjective, Objective, Assessment, Plan, and Others.
The datasets utilized were:
- Discharge: Discharge summaries from the i2b2 2010 challenge, representing Partners Healthcare and Beth Israel Deaconess.
- THYME: Colorectal clinical notes from the Mayo Clinic.
- Progress: ICU progress notes sourced from the MIMIC-III database.
Results and Discussion
The findings highlight both the challenges of cross-domain transfer and the effectiveness of adaptive pretraining:
- The Transfer Gap: When models were moved from an in-domain to a cross-domain setting without adaptation, the average F1 score plummeted from 0.977 to 0.618, confirming the severity of the documentation style mismatch.
- The Synergistic Effect: Unsupervised pretraining (DAPT/TAPT) alone did not resolve the transfer issue in a "cold start" scenario (zero labeled samples). However, it acted as a powerful force multiplier when combined with even minimal labeled data. With just 10–50 labeled target samples, the average F1 score rose from 0.756 to 0.808.
- Quantifying the Benefit: On average, the DAPT+TAPT pipeline provided a performance boost equivalent to 50.2 manually annotated in-domain samples. This provides a clear metric for institutions to weigh the cost of computational pretraining against the cost of expert manual labeling.
- Diminishing Returns of TAPT: Interestingly, TAPT provided only marginal gains over DAPT in this clinical context. This suggests that the broader "clinical language" domain captured by DAPT is often sufficient for section classification tasks.
Conclusion and Limitations
Continued pretraining offers a viable path for deploying clinical NLP models across different institutions with minimal manual labeling effort. However, the authors note that the computational infrastructure required for pretraining may be a barrier for smaller clinics. In such cases, manually labeling a small set of notes might be more efficient. Future research should focus on multi-center training to improve inherent model robustness and explore more granular section labels beyond the high-level SOAP framework.
Summary updated