Abstract
Despite the significant potential of Foundation Models (FMs) in medical imaging, their application to prognosis prediction remains challenging due to data scarcity, class imbalance, and task complexity. This study introduces a structured benchmark to assess the robustness and efficiency of transfer learning strategies for FMs compared with convolutional neural networks (CNNs) in predicting COVID-19 patient outcomes from chest X-rays. By evaluating models across four datasets and various fine-tuning regimes, the research provides empirical guidance for AI deployment in clinical workflows. Results indicate that while CNNs remain dependable for low-resource scenarios, FMs with Parameter-Efficient Fine-Tuning (PEFT) achieve competitive performance as data availability increases.
Introduction
In medical imaging, research typically focuses on diagnosis (identifying a condition) or segmentation (outlining structures). However, prognosis—predicting future outcomes such as mortality, ICU admission, or disease severity—is critical for clinical decision-making. Prognostic tasks are particularly challenging due to "the three horsemen" of medical data: extreme data scarcity, severe class imbalance, and high task complexity. While large-scale Foundation Models have revolutionized general computer vision, their effectiveness in these constrained medical scenarios was previously unmapped. This study systematically evaluates how different architectures and adaptation strategies handle realistic clinical constraints.
Methodology
The researchers conducted an extensive evaluation comparing two primary architectural families:
- Convolutional Neural Networks (CNNs): Standard models like ResNet and DenseNet, pretrained on ImageNet.
- Foundation Models (FMs): Large-scale models such as DINOv2 and CLIP, including domain-specific variants like MedCLIP and BioMedCLIP.
The study tested several adaptation strategies as illustrated in the framework above:
- Full Fine-Tuning (FFT): Updating all model parameters.
- Linear Probing (LP): Training only a new classification head while freezing the backbone.
- Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA (Low-Rank Adaptation) and BitFit that update less than 1% of parameters.
These were evaluated using four COVID-19 Chest X-ray (CXR) datasets: AIforCOVID, COVID-19-AR, CoCross, and Stony Brook COVID-19 (in NY_small and NY_all versions). Evaluation metrics focused on Matthews Correlation Coefficient (MCC) and Precision-Recall AUC (PR-AUC) to provide a more honest assessment of model performance on imbalanced medical data than standard accuracy.
Key Insights and Results
The benchmark revealed several critical findings regarding model selection and adaptation:
- CNNs as the "Reliable Old Guard": For extremely small or highly imbalanced datasets, CNNs with full fine-tuning remain the most dependable choice. Their strong inductive biases help prevent overfitting when data is scarce.
- Generalists vs. Specialists: Surprisingly, general-purpose FMs (e.g., DINOv2) often matched or outperformed biomedical-specific models. The massive scale of general pretraining appears more beneficial than smaller-scale domain-specific pretraining for these prognostic tasks.
- PEFT Scalability and Sensitivity: PEFT methods (LoRA, BitFit) are highly efficient and achieve competitive results on larger datasets like
NY_all. However, their performance degrades significantly under severe class imbalance unless mitigation strategies are employed. - Few-Shot Stability: In extreme few-shot scenarios (2 to 32 examples per class), Linear Probing proved more stable than complex PEFT methods, which tended to overfit the small training sets and fail on imbalanced test data.
Conclusion
There is no "one-size-fits-all" strategy for medical prognosis. For rare diseases with very limited data, traditional CNNs are preferred. Conversely, for conditions with sufficient data, Foundation Models paired with PEFT offer a superior, more efficient path. Future work should extend this benchmark to longitudinal data and other imaging modalities like CT or MRI to further refine clinical AI deployment.
Summary updated