Abstract
Pre-trained transformers are often fine-tuned to aid clinical decision-making using limited clinical notes. Model interpretability is crucial, especially in high-stakes domains like medicine, to establish trust and ensure safety, which requires human engagement. We introduce SUFO, a systematic framework that enhances interpretability of fine-tuned transformer feature spaces. SUFO utilizes a range of analytic and visualization techniques, including Supervised probing, Unsupervised similarity analysis, Feature dynamics, and Outlier analysis to address key questions about model trust and interpretability. We conduct a case study investigating the impact of pre-training data where we focus on real-world pathology classification tasks, and validate our findings on MedNLI. We evaluate five 110M-sized pre-trained transformer models, categorized into general-domain (BERT, TNLR), mixed-domain (BioBERT, Clinical BioBERT), and domain-specific (PubMedBERT) groups. Our SUFO analyses reveal that: (1) while PubMedBERT, the domain-specific model, contains valuable information for fine-tuning, it can overfit to minority classes when class imbalances exist. In contrast, mixed-domain models exhibit greater resistance to overfitting, suggesting potential improvements in domain-specific model robustness; (2) in-domain pre-training accelerates feature disambiguation during fine-tuning; and (3) feature spaces undergo significant sparsification during this process, enabling clinicians to identify common outlier modes among fine-tuned models as demonstrated in this paper. These findings showcase the utility of SUFO in enhancing trust and safety when using transformers in medicine, and we believe SUFO can aid practitioners in evaluating fine-tuned language models for other applications in medicine and in more critical domains.
Introduction
Transformer-based models are increasingly deployed to assist clinicians in high-stakes decision-making by processing unstructured clinical notes. However, these models are frequently treated as "black boxes." While standard performance metrics like F1 scores provide a measure of accuracy, they offer little insight into the internal evolution of feature spaces during fine-tuning or the underlying reasons for model failure. This lack of interpretability poses a significant barrier to the adoption of AI in medicine, where trust and safety are paramount.
This study introduces the SUFO framework to investigate how pre-training data—ranging from general-domain (Wikipedia) to domain-specific (PubMed) sources—influences the reliability and interpretability of features learned for clinical tasks.
The SUFO Framework
The researchers propose SUFO, a diagnostic framework consisting of four analytical lenses:
- Supervised Probing: Evaluates the task-relevant information present in frozen model features prior to fine-tuning by training a simple linear layer.
- Unsupervised Similarity Analysis: Employs Representational Similarity Analysis (RSA) to quantify the reconfiguration of internal feature spaces during the fine-tuning process.
- Feature Dynamics: Uses Principal Component Analysis (PCA) to visualize the clustering of data points across layers and training epochs, identifying when and where models disambiguate classes.
- Outlier Analysis: Identifies data points that deviate from learned clusters, followed by domain-expert evaluation to characterize model failure modes.
Experimental Setup
The framework was applied to a private corpus of 2,907 structured Prostate Cancer Pathology Reports from UCSF. Tasks included classifying Gleason grades, margin status, and seminal vesicle invasion—all characterized by significant class imbalance. Findings were validated on the public MedNLI clinical inference dataset. Five 110M-parameter models were compared: BERT, TNLR (general), BioBERT, Clinical BioBERT (mixed), and PubMedBERT (domain-specific).
Key Insights and Results
Supervised Probing and the PubMedBERT Paradox
Supervised probing revealed that PubMedBERT, trained from scratch on medical text, often possessed the most relevant "raw" features for pathology tasks before fine-tuning began.
However, this initial advantage did not always translate to superior final performance. On highly imbalanced tasks like "Primary Gleason," PubMedBERT’s F1 score dropped to 0.770, while mixed-domain models like Clinical BioBERT maintained a high 0.959. This suggests that while domain-specific pre-training provides high-quality features, it may also increase susceptibility to overfitting on imbalanced clinical data.
Feature Space Reconfiguration
Unsupervised similarity analysis (RSA) showed how different models reconfigure their internal representations. General-domain models like TNLR often underwent more drastic changes in their early layers compared to domain-specific models.
Sparsity and Disambiguation
A striking finding was the "sparsity phenomenon": after fine-tuning, the first two principal components explained approximately 95% of the variance in the feature space. This indicates that fine-tuning collapses complex language data into a low-dimensional structure optimized for the specific task. Furthermore, in-domain pre-training accelerated this process; medical-specific models began disambiguating features as early as epoch 6, whereas general BERT required until epoch 9.
Robustness and Outlier Analysis
Mixed-domain models (BioBERT, Clinical BioBERT) emerged as the most robust performers, likely because general-domain data acts as a regularizer. Outlier analysis, validated by a urologist, showed that Clinical BioBERT was particularly effective at identifying "Hard Outliers"—cases with missing or ambiguous information—rather than simply failing on them.
Conclusion
The SUFO framework demonstrates that evaluating clinical models requires looking beyond top-line accuracy. By illuminating the internal dynamics of transformer feature spaces, SUFO provides a pathway for building more trustworthy and robust AI systems for medicine. Future work will explore whether these dynamics hold for larger-scale models and generative clinical tasks.
Summary updated