An Investigation into the Effects of Pre-training Data Distributions for Pathology Report Classification

Aliyah R. Hsu, Yeshwanth Cherapanamjeri, Briton Park, Tristan Naumann, Anobel Y. Odisho, Bin Yu

Abstract

Large Language Models (LLMs) are increasingly integrated into clinical workflows, yet their internal decision-making processes remain opaque. This study introduces SUFO, a diagnostic framework designed to analyze the internal feature dynamics of transformer models during fine-tuning on clinical notes. By comparing general-domain, mixed-domain, and domain-specific models, we demonstrate that while domain-specific pre-training provides relevant raw features, mixed-domain models (e.g., Clinical BioBERT) offer superior stability and performance, particularly in imbalanced clinical datasets. Our findings suggest that mixed-domain pre-training facilitates faster feature disambiguation and greater robustness against overfitting.

Introduction

The deployment of LLMs in high-stakes medical fields like oncology necessitates a move beyond simple accuracy metrics toward a deeper understanding of model interpretability. A critical gap exists in understanding how pre-training data distributions—ranging from general web text to specialized medical literature—influence the internal "feature space" during task-specific fine-tuning. Without this insight, developing clinical AI that is both reliable and safe remains a challenge. This paper addresses this gap by examining how different pre-training strategies affect the evolution of internal logic during the classification of prostate cancer pathology reports.

Methodology

The researchers developed the SUFO framework, which utilizes four analytical pillars:

  1. Supervised Probing: Assessing the utility of features prior to task-specific training.
  2. Unsupervised Similarity Analysis: Employing Representational Similarity Analysis (RSA) to track layer-wise conceptual shifts during fine-tuning.
  3. Feature Dynamics: Visualizing data clustering across training epochs and layers using Principal Component Analysis (PCA).
  4. Outlier Analysis: Investigating data points that deviate from established clusters with expert clinical input.

The study evaluated five models: General-domain (BERT, TNLR), Mixed-domain (BioBERT, Clinical BioBERT), and Domain-specific (PubMedBERT). These were tested on a private dataset of 2,907 UCSF prostate cancer pathology reports and validated using the public MedNLI dataset.

Results and Analysis

Performance and Robustness

As shown in the performance evaluation, mixed-domain models (BioBERT and Clinical BioBERT) achieved the highest average F1 scores, approximately 0.95-0.96. A key finding was the "PubMedBERT Paradox": while PubMedBERT possesses the most relevant raw features for medical tasks, it proved highly unstable on imbalanced data. For instance, on the Path-PG task, PubMedBERT’s performance dropped to an F1 of 0.770, whereas Clinical BioBERT maintained a robust 0.959.

Table 1: F1 performance metrics across four pathology classification tasks, highlighting the superior average performance of mixed-domain models.

Feature Space Dynamics and Similarity

Analysis of the models' internal feature spaces post-fine-tuning revealed extreme sparsity, with the first two principal components explaining roughly 95% of the data variance. This indicates that models aggressively discard non-essential information to focus on task-specific features.

Figure 1: Layer-wise RSA similarity for the Primary Gleason task, showing the divergence of fine-tuned representations from their base models.

Unsupervised similarity analysis (RSA) demonstrated that mixed-domain models disambiguate features faster than their general-domain counterparts. Clinical models began to clearly separate data classes around Epoch 6, whereas general models like BERT required until Epoch 9 to reach comparable clarity.

Discussion and Conclusion

The SUFO framework reveals that pre-training data distribution is a primary determinant of model robustness in clinical settings. Mixed-domain pre-training appears to provide a "sweet spot," combining the broad linguistic foundations of general text with the specialized knowledge of medical corpora to prevent the overfitting observed in purely domain-specific models. Furthermore, the identification of outliers through SUFO serves as a valuable tool for clinical quality control, highlighting inconsistencies or boundary cases in medical documentation.

Future research should explore whether these feature-space dynamics persist in larger generative models and investigate optimal data mixing strategies for complex tasks like medical reasoning and summarization.

Summary updated

BibTeX
@inproceedings{hsu2024investigation,
  author = {Aliyah R. Hsu and Yeshwanth Cherapanamjeri and Briton Park and Tristan Naumann and Anobel Y. Odisho and Bin Yu},
  title = {{An Investigation into the Effects of Pre-training Data Distributions for Pathology Report Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=k581sTMyPt},
}
Figure