Benchmarking foundation models and parameter-efficient fine-tuning for prognosis prediction in medical imaging

Filippo Ruffini, Elena Mulero Ayllon, Linlin Shen, Paolo Soda, Valerio Guarrasi

TL;DR

This paper presents the first comprehensive benchmark comparing Foundation Models (FMs) and traditional Convolutional Neural Networks (CNNs) for medical prognosis—predicting patient outcomes rather than just diagnosing disease. The researchers found that while CNNs remain the "reliable old guard" for extremely small or imbalanced datasets, FMs paired with Parameter-Efficient Fine-Tuning (PEFT) offer superior scalability and efficiency as data availability increases.

What Problem Does This Paper Address?

In medical imaging, most AI research focuses on diagnosis (identifying what is there now) or segmentation (outlining organs). However, prognosis—predicting future outcomes like mortality, ICU admission, or disease severity—is arguably more critical for clinical decision-making.

Prognosis tasks are notoriously difficult because they suffer from "the three horsemen" of medical data: extreme data scarcity, severe class imbalance (e.g., many more survivors than deaths), and high task complexity. While large-scale Foundation Models (like those based on Transformers) have revolutionized general computer vision, their effectiveness in these constrained, high-stakes medical prognostic scenarios remained largely unmapped. This paper fills that gap by systematically testing how different model architectures and fine-tuning strategies handle realistic clinical constraints.

Methods

The authors conducted a massive evaluation (requiring ~3,200 GPU-hours) comparing two architectural families:

  1. CNNs: Standard models like ResNet and DenseNet, pretrained on ImageNet.
  2. Foundation Models (FMs): Large-scale models like DINOv2 and CLIP, including domain-specific versions like MedCLIP and BioMedCLIP.

They tested several adaptation strategies:

  • Full Fine-Tuning (FFT): Updating every single parameter in the model.
  • Linear Probing (LP): Freezing the model and only training a new "head" on top.
  • PEFT (Parameter-Efficient Fine-Tuning): Strategies like LoRA (Low-Rank Adaptation) and BitFit that only update a tiny fraction (often <1%) of the model's parameters.

These were evaluated under both "full-data" regimes and Few-Shot Learning (FSL) scenarios, where the model is trained on as few as 2 to 32 examples per class.

Datasets

The study utilized four publicly available COVID-19 Chest X-ray (CXR) datasets, chosen for their diversity in tasks and sample sizes:

  • AIforCOVID: Focused on severity and mortality (1,585 patients).
  • COVID-19-AR: Focused on ICU admission in rural populations (99 samples).
  • CoCross: ICU outcomes for longitudinal scans (387 samples).
  • Stony Brook COVID-19: A large-scale dataset used in two versions—NY_small (one scan per patient) and NY_all (all scans during hospitalization, totaling 13,639 images).

Key Insights and Ideas

  • The "Reliable Old Guard" vs. The "Scalable Giant": CNNs with full fine-tuning are still the most dependable choice for tiny, highly imbalanced datasets. Their strong inductive biases help them avoid overfitting when data is scarce.
  • Generalists Beat Specialists: Surprisingly, general-purpose FMs (like DINOv2) often outperformed or matched biomedical-specific models (like MedCLIP). This suggests that the massive scale of general pretraining might be more valuable than smaller-scale domain-specific pretraining for prognosis.
  • PEFT is Sensitive to Imbalance: While PEFT (especially LoRA and BitFit) is incredibly efficient, its performance drops significantly under severe class imbalance. However, when the data is balanced, PEFT allows FMs to bridge the gap and even outperform CNNs.
  • Linear Probing is the Few-Shot Winner: In extreme few-shot scenarios, simply training a linear head (Linear Probing) was often more stable than more complex PEFT methods, which tended to overfit the small, balanced training sets and fail on imbalanced test data.

Key Results

  • Top Performers: DINOv2 and PubMedCLIP emerged as the strongest FMs across most tasks.
  • Robustness: In the NY_all dataset (the largest tested), PEFT methods achieved competitive results with Full Fine-Tuning while updating only a fraction of the parameters.
  • Metric Choice: The authors emphasized using Matthews Correlation Coefficient (MCC) and PR-AUC instead of standard Accuracy or ROC-AUC, as these metrics provide a much more honest assessment of model performance on imbalanced medical data.

Limitations and Future Work

The authors acknowledge several key limitations:

  1. Single-Image Focus: The current benchmark only looks at individual images, ignoring the temporal "story" of a patient's progression found in longitudinal scans.
  2. Modality Constraint: The study is limited to Chest X-rays. Future work should expand to CT, MRI, and pathology slides.
  3. The "Black Box" of PEFT: While PEFT is efficient, we still don't fully understand why certain configurations (like specific ranks in LoRA) work better for medical tasks than others.

The paper concludes that there is no "one-size-fits-all" strategy. For a new, rare disease with 50 images, stick with a CNN. For a well-documented condition with thousands of images, an FM with PEFT is likely the superior, more efficient path forward.

Summary updated

Abstract

Despite the significant potential of Foundation Models (FMs) in medical imaging, their application to prognosis prediction remains challenging due to data scarcity, class imbalance, and task complexity, which limit their clinical adoption. This study introduces the first structured benchmark to assess the robustness and efficiency of transfer learning strategies for FMs compared with convolutional neural networks (CNNs) in predicting COVID-19 patient outcomes from chest X-rays. The goal is to systematically compare finetuning strategies, both classical and parameter efficient, under realistic clinical constraints related to data scarcity and class imbalance, offering empirical guidance for AI deployment in clinical workflows. Four publicly available COVID-19 chest X-ray datasets were used, covering mortality, severity, and ICU admission, with varying sample sizes and class imbalances. CNNs pretrained on ImageNet and FMs pretrained on general or biomedical datasets were adapted using full finetuning, linear probing, and parameter-efficient methods. Models were evaluated under full data and few shot regimes using the Matthews Correlation Coefficient (MCC) and Precision Recall AUC (PR-AUC), with cross validation and class weighted losses. CNNs with full fine-tuning performed robustly on small, imbalanced datasets, while FMs with Parameter-Efficient Fine-Tuning (PEFT), particularly LoRA and BitFit, achieved competitive results on larger datasets. Severe class imbalance degraded PEFT performance, whereas balanced data mitigated this effect. In few-shot settings, FMs showed limited generalization, with linear probing yielding the most stable results. No single fine-tuning strategy proved universally optimal: CNNs remain dependable for low-resource scenarios, whereas FMs benefit from parameter-efficient methods when data are sufficient.
BibTeX
@inproceedings{ruffini2026benchmarking,
  author = {Filippo Ruffini and Elena Mulero Ayllon and Linlin Shen and Paolo Soda and Valerio Guarrasi},
  title = {{Benchmarking foundation models and parameter-efficient fine-tuning for prognosis prediction in medical imaging}},
  booktitle = {Comput. Methods Programs Biomed},
  year = {2026},
  url = {https://openreview.net/forum?id=iujbkgH2UL},
}

Loading related content...

Figure