Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging

Filippo Ruffini, Elena Mulero Ayllon, Linlin Shen, Paolo Soda, Valerio Guarrasi

Abstract

Despite the significant potential of Foundation Models (FMs) in medical imaging, their application to prognosis prediction remains challenging due to data scarcity, class imbalance, and task complexity. This study introduces a structured benchmark to assess the robustness and efficiency of transfer learning strategies for FMs compared with convolutional neural networks (CNNs) in predicting COVID-19 patient outcomes from chest X-rays. By evaluating models across four datasets and various fine-tuning regimes, the research provides empirical guidance for AI deployment in clinical workflows. Results indicate that while CNNs remain dependable for low-resource scenarios, FMs with Parameter-Efficient Fine-Tuning (PEFT) achieve competitive performance as data availability increases.

Introduction

In medical imaging, research typically focuses on diagnosis (identifying a condition) or segmentation (outlining structures). However, prognosis—predicting future outcomes such as mortality, ICU admission, or disease severity—is critical for clinical decision-making. Prognostic tasks are particularly challenging due to "the three horsemen" of medical data: extreme data scarcity, severe class imbalance, and high task complexity. While large-scale Foundation Models have revolutionized general computer vision, their effectiveness in these constrained medical scenarios was previously unmapped. This study systematically evaluates how different architectures and adaptation strategies handle realistic clinical constraints.

Methodology

The researchers conducted an extensive evaluation comparing two primary architectural families:

  1. Convolutional Neural Networks (CNNs): Standard models like ResNet and DenseNet, pretrained on ImageNet.
  2. Foundation Models (FMs): Large-scale models such as DINOv2 and CLIP, including domain-specific variants like MedCLIP and BioMedCLIP.
Overview of the benchmarking framework, including models, fine-tuning strategies, datasets, and evaluation metrics.

The study tested several adaptation strategies as illustrated in the framework above:

  • Full Fine-Tuning (FFT): Updating all model parameters.
  • Linear Probing (LP): Training only a new classification head while freezing the backbone.
  • Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA (Low-Rank Adaptation) and BitFit that update less than 1% of parameters.

These were evaluated using four COVID-19 Chest X-ray (CXR) datasets: AIforCOVID, COVID-19-AR, CoCross, and Stony Brook COVID-19 (in NY_small and NY_all versions). Evaluation metrics focused on Matthews Correlation Coefficient (MCC) and Precision-Recall AUC (PR-AUC) to provide a more honest assessment of model performance on imbalanced medical data than standard accuracy.

Key Insights and Results

The benchmark revealed several critical findings regarding model selection and adaptation:

  • CNNs as the "Reliable Old Guard": For extremely small or highly imbalanced datasets, CNNs with full fine-tuning remain the most dependable choice. Their strong inductive biases help prevent overfitting when data is scarce.
  • Generalists vs. Specialists: Surprisingly, general-purpose FMs (e.g., DINOv2) often matched or outperformed biomedical-specific models. The massive scale of general pretraining appears more beneficial than smaller-scale domain-specific pretraining for these prognostic tasks.
  • PEFT Scalability and Sensitivity: PEFT methods (LoRA, BitFit) are highly efficient and achieve competitive results on larger datasets like NY_all. However, their performance degrades significantly under severe class imbalance unless mitigation strategies are employed.
  • Few-Shot Stability: In extreme few-shot scenarios (2 to 32 examples per class), Linear Probing proved more stable than complex PEFT methods, which tended to overfit the small training sets and fail on imbalanced test data.

Conclusion

There is no "one-size-fits-all" strategy for medical prognosis. For rare diseases with very limited data, traditional CNNs are preferred. Conversely, for conditions with sufficient data, Foundation Models paired with PEFT offer a superior, more efficient path. Future work should extend this benchmark to longitudinal data and other imaging modalities like CT or MRI to further refine clinical AI deployment.

Summary updated

Abstract

Despite the significant potential of Foundation Models (FMs) in medical imaging, their application to prognosis prediction remains challenging due to data scarcity, class imbalance, and task complexity, which limit their clinical adoption. This study introduces the first structured benchmark to assess the robustness and efficiency of transfer learning strategies for FMs compared with convolutional neural networks (CNNs) in predicting COVID-19 patient outcomes from chest X-rays. The goal is to systematically compare finetuning strategies, both classical and parameter efficient, under realistic clinical constraints related to data scarcity and class imbalance, offering empirical guidance for AI deployment in clinical workflows. Four publicly available COVID-19 chest X-ray datasets were used, covering mortality, severity, and ICU admission, with varying sample sizes and class imbalances. CNNs pretrained on ImageNet and FMs pretrained on general or biomedical datasets were adapted using full finetuning, linear probing, and parameter-efficient methods. Models were evaluated under full data and few shot regimes using the Matthews Correlation Coefficient (MCC) and Precision Recall AUC (PR-AUC), with cross validation and class weighted losses. CNNs with full fine-tuning performed robustly on small, imbalanced datasets, while FMs with Parameter-Efficient Fine-Tuning (PEFT), particularly LoRA and BitFit, achieved competitive results on larger datasets. Severe class imbalance degraded PEFT performance, whereas balanced data mitigated this effect. In few-shot settings, FMs showed limited generalization, with linear probing yielding the most stable results. No single fine-tuning strategy proved universally optimal: CNNs remain dependable for low-resource scenarios, whereas FMs benefit from parameter-efficient methods when data are sufficient.
BibTeX
@inproceedings{ruffini2026benchmarking,
  author = {Filippo Ruffini and Elena Mulero Ayllon and Linlin Shen and Paolo Soda and Valerio Guarrasi},
  title = {{Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging}},
  booktitle = {Comput. Methods Programs Biomed},
  year = {2026},
  url = {https://openreview.net/forum?id=iujbkgH2UL},
}
Figure