Revealing Ethical Risks in Pathology Foundation Models: A Quantitative Perspective

Liansheng Wang, Weiping Lin, Shen Liu, Rong Zhu, Yixuan Lin, Baoshun WangResearch Square 2025 · DOI

TL;DR

While Pathology Foundation Models (PFMs) have revolutionized diagnostic accuracy, they carry hidden ethical baggage. This paper introduces QERA (Quantitative Ethical Risk Assessment), the first framework to prove that these models inadvertently leak sensitive patient data, rely on "spurious correlations" like hospital-specific staining, and exhibit performance disparities across demographic groups.

What Problem Does This Paper Address?

In the world of digital pathology, "Foundation Models" (like UNI, Virchow, and GigaPath) are the new gold standard. These massive models are pretrained on millions of tissue images to extract features that can be used for everything from cancer grading to gene mutation prediction.

However, there is a catch: most researchers evaluate these models solely on accuracy. This paper argues that accuracy is a dangerous metric if the model is "cheating" by using non-diagnostic information. For example, if a model learns to identify a specific hospital's unique slide-staining style rather than the actual cancer cells, it might look accurate in the lab but fail miserably—or unfairly—in a real-world clinical setting. The authors address the lack of quantitative tools to measure these ethical risks: privacy leakage, clinical reliability, and group fairness.

Methods

The authors developed the QERA framework to stress-test three state-of-the-art PFMs (UNI, Prov-GigaPath, and Virchow). Their approach involves:

  1. Probing for Privacy: They trained "lightweight" classifiers on top of the PFM embeddings to see if they could "guess" a patient’s age, gender, race, or the medical institution where the slide was processed.
  2. OOD Stress Testing: To measure reliability, they designed three Out-of-Distribution (OOD) scenarios:
  3. OOD1: Training and testing on different institutions.
  4. OOD2: Introducing a "spurious correlation" (e.g., Institution A only provides Class 0, Institution B only provides Class 1) to see if the model relies on the institution rather than the pathology.
  5. OOD3: The most stringent test, where the institution-label correlation is completely inverted during testing.
  6. Fairness Auditing: They measured performance gaps across demographic subgroups and calculated the Coefficient of Variation (CV) to quantify how much accuracy fluctuated between different hospitals.

Datasets

The study utilized 26 curated datasets derived primarily from public sources like The Cancer Genome Atlas (TCGA).

  • Privacy/Fairness: Used cohorts like TCGA-LUAD (Lung), KIRC (Kidney), and COAD (Colorectal) with associated clinical metadata (age, race, gender).
  • Institutional Data: Created the TCGA-UT and OA-INS datasets to specifically track which medical centers provided which slides.
  • Tasks: The evaluation spanned patch classification, whole-slide image (WSI) classification, survival analysis, and image retrieval.

Key Insights and Ideas

  • The "SSL Trap": Most PFMs use Self-Supervised Learning (SSL), which encourages the model to capture every difference between images. Unfortunately, this means the models are highly efficient at capturing "noise" like institutional staining protocols, which they then use as a shortcut for diagnosis.
  • Stain Normalization is Not a Silver Bullet: A common fix in pathology is "stain normalization" (making all slides look like they were dyed the same way). The authors found this helps slightly but fails to remove the underlying institutional signatures encoded in the models.
  • Supervision as a Shield: Models trained with some form of guidance—like CONCH, which uses image-text matching—showed significantly lower ethical risks. This suggests that "blind" pretraining on massive data is more prone to bias than targeted, multi-modal training.
  • The Hidden Leak: The fact that a simple classifier can predict a patient's race with over 90% accuracy just from a tissue embedding is a major red flag for data privacy and potential "algorithmic profiling."

Key Results

  • Privacy Leakage: Classifiers achieved non-trivial accuracy in predicting sensitive attributes. For example, race prediction accuracy reached ~87-93% across different models (Extended Table 2).
  • Reliability Collapse: In the "adversarial" OOD3 setting, performance plummeted. For the LUNG-tile dataset, UNI’s F1-score dropped from a baseline of 0.6622 to 0.3536 (Extended Table 4), proving the model was relying heavily on institutional shortcuts.
  • Fairness Gaps: The authors found consistent performance disparities. In some kidney cancer tasks (RCC-tile), the accuracy gap between racial groups was as high as 15.9% (Extended Table 13).
  • Mitigation: Applying LoRA (Low-Rank Adaptation) finetuning significantly improved OOD performance, suggesting that a small amount of supervised "re-tuning" can help steer models away from ethical risks.

Limitations and Future Work

  • Dataset Bias: The study relies heavily on TCGA data. While diverse, TCGA has its own inherent biases in how samples were collected and labeled.
  • The "Black Box" of SSL: While the paper identifies that SSL captures non-diagnostic features, more work is needed to develop new SSL objectives that explicitly "disentangle" pathology from institutional noise.
  • Clinical Translation: The authors call for the QERA framework to be integrated into the standard "auditing" pipeline before any PFM is cleared for use in a hospital, moving the field toward "Ethical-by-Design" AI.

Summary updated

Abstract

<title>Abstract</title> Pathology foundation models (PFMs) are large-scale pretrained models tailored for computational pathology, substantially advancing the development of downstream models across a wide array of diagnostic tasks. However, their clinical deployment raises ethical concerns, including privacy leakage, reliability in diverse clinical settings, and fairness across patient subgroups. These risks remain underexplored and insufficiently quantified, yet addressing them is critical to enable the safe translation of PFMs from research to clinical practice. In this study, we propose the first quantitative framework to systematically assess ethical risks in PFMs. We focus on three critical dimensions: potential leakage of patient-sensitive attributes, reliance on non-diagnostic features that compromise reliability, and subgroup disparities in performance. To this end, we design task-specific experiments across a wide range of datasets and clinical scenarios, providing a quantitative and auditable protocol for ethical risk evaluation. The results show that while current PFMs demonstrate strong performance, they still exhibit challenges in patient privacy, clinical reliability, and group fairness. We further explore the underlying causes of these ethical risks in PFMs, substantiate our findings with empirical evidence, and provide insights into strategies for their mitigation. This work provides the first quantitative and systematic assessment of ethical risks in PFMs. Our findings highlight the urgent need for ethical safeguards in PFMs and offer actionable insights for building more trustworthy and clinically robust PFMs. We will release these evaluation results and provide the assessment framework as a toolkit to support the development, auditing, and deployment of PFMs in both research and translational settings.
BibTeX
@inproceedings{wang2025revealing,
  author = {Liansheng Wang and Weiping Lin and Shen Liu and Rong Zhu and Yixuan Lin and Baoshun Wang},
  title = {{Revealing Ethical Risks in Pathology Foundation Models: A Quantitative Perspective}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7644824/v1},
}
Figure