When Foundation Models Disagree: Representational Heterogeneity for Calibrated Uncertainty in Whole-Slide Image Classification

Yekai Shen, Yuqian Wang, Zishun Liao, Rongsheng Wang, Wenting Chen, S. Kevin Zhou, Qingsong YaoarXiv 2026 · arXiv

In the high-stakes world of digital pathology, AI models are increasingly used to classify gigapixel Whole-Slide Images (WSIs) for tumor subtyping and biomarker prediction. While modern "Foundation Models" (FMs) provide incredibly rich representations of tissue, they suffer from a critical flaw: they are often overconfident, even when they are wrong.

A new paper, "When Foundation Models Disagree," introduces CALIFUSE, a framework that turns model disagreement from a nuisance into a powerful signal for clinical safety.

TL;DR

CALIFUSE is a training-free framework that estimates uncertainty by measuring how much different pathology foundation models disagree on a single slide. By combining slide-level consensus with fine-grained "evidence agreement" across tissue chunks, it significantly improves model calibration and error detection without sacrificing classification accuracy.

What Problem Does This Paper Address?

Current pathology AI pipelines typically follow a two-step process: a frozen Foundation Model (FM) encodes tissue patches into features, and a Multiple Instance Learning (MIL) head aggregates those features into a slide-level diagnosis.

The problem is that most uncertainty estimation methods only look at instability within a single feature space (e.g., using Monte Carlo dropout). However, different FMs—trained on different datasets with different architectures—often "see" the same slide differently. A slide might look straightforward to one model but ambiguous to another.

If we only use one model, we miss the uncertainty introduced by the choice of the model itself. In a clinical setting, we need "calibrated" uncertainty: if a model says it’s 90% sure, it should be right 90% of the time. If it isn't, we can't safely triage difficult cases for human review.

Methods

CALIFUSE (Calibration-oriented Fusion) operates on the principle of representational heterogeneity. Instead of just averaging predictions to boost accuracy, it treats the "epistemic dissonance" between models as an explicit cue for uncertainty.

The framework consists of two main branches:

  1. Decision Branch (Slide-level): It looks at the final labels predicted by a pool of $N$ models (e.g., UNI, Virchow2, GigaPath). It uses "consensus-conditioned pooling" to determine a final prediction and calculates a confidence score based on how much the models agreed.
  2. Evidence Branch (Chunk-level): This is the "fine-grained" check. The slide is divided into chunks, and the framework calculates the Jensen-Shannon (JS) divergence across the predictive distributions of all models for every chunk. If models agree on the final label but disagree on which parts of the tissue provide the evidence, the confidence score drops.

The final confidence is a simple weighted combination of these two signals. Crucially, this is training-free; it doesn't require training a complex fusion module, making it easy to plug into existing workflows.

Overview of the CALIFUSE framework, showing the Decision and Evidence branches.

Datasets

The authors evaluated CALIFUSE across seven diverse WSI classification tasks:

  • TCGA Benchmarks: Lung (BLCA), Breast (BRCA-TNBC), Kidney, Sarcoma, and Brain (GBM/LGG) tasks covering both tumor subtyping and biomarker prediction.
  • EBRAINS Digital Brain Tumour Atlas: Two tasks (11-class and 30-class) representing highly complex, fine-grained classification challenges.

Key Insights and Ideas

  • Disagreement is Informative: In traditional ensembling, disagreement is often "smoothed away." CALIFUSE argues that when strong models part ways, it’s rarely noise; it’s a clinically meaningful sign that a case is borderline, atypical, or underrepresented in training data.
  • The "Evidence" Matters: Two models might reach the same conclusion (e.g., "Cancer") but point to different regions of the slide as the reason. CALIFUSE captures this "evidence inconsistency" through JS divergence, catching cases where a slide-level vote might look deceptively stable.
  • Calibration vs. Accuracy: While naive averaging (ensembling) often improves accuracy, it rarely improves calibration. CALIFUSE specifically targets the "Reliability Gap," ensuring that the reported confidence actually reflects the true risk of error.

Key Results

The researchers compared CALIFUSE against the "Oracle" (the best-performing single model for a given metric) and a "Naive Ensemble" (simple probability averaging).

  • Superior Calibration: CALIFUSE improved the Expected Calibration Error (ECE) by an average of 44% over the best single-feature baseline.
  • Better Error Detection: It excelled at ranking incorrect predictions, showing a 34% improvement in Area Under the Risk-Coverage curve (AURC) and a 16% boost in AUROC for error detection.
  • Maintained Accuracy: Despite the focus on uncertainty, the framework remained competitive in balanced accuracy, often matching or exceeding the strongest multi-model baselines.

As shown in the reliability diagrams below, CALIFUSE (pink) tracks the ideal diagonal much more closely than a standard ensemble (blue), which tends to be overconfident.

Reliability diagrams showing CALIFUSE (pink) vs. Naive Ensemble (blue).

Limitations and Future Work

  • Computational Cost: The main drawback is the need to run $N$ different foundation models and their respective MIL heads. While the fusion itself is "training-free," the inference cost scales linearly with the number of models in the pool.
  • Hyperparameter Sensitivity: The current work uses a fixed weight ($\lambda = 0.5$) to balance the decision and evidence branches. The authors suggest that tuning this per task or adding a post-hoc calibration layer (like temperature scaling) could further refine the results.
  • Future Directions: The authors propose extending this to "selective referral" systems, where the AI automatically flags high-uncertainty slides for a pathologist's priority queue.

Summary updated

Abstract

Pathology foundation models supply strong representations for whole-slide image classification, yet the confidence attached to their predictions remains hard to trust. Prevailing uncertainty estimators act within a single feature space and thereby overlook the uncertainty that the choice of foundation-model representation itself induces. This work introduces CALIFUSE, a calibration-oriented framework that fuses several pathology foundation models for the specific purpose of uncertainty estimation. Departing from accuracy-driven fusion, CALIFUSE reads representational heterogeneity across encoders as an explicit confidence cue. Slide-level decision consensus is coupled with chunk-level evidence agreement quantified through Jensen-Shannon divergence, so that both prediction stability and fine-grained evidence consistency inform the estimate. Foundation-model encoders stay frozen. A task-specific nnMIL classifier is still trained for each FM; CALIFUSE then estimates confidence at inference without training an extra fusion module. Evaluated on seven WSI classification tasks under several foundation-model combinations, the framework sharpens calibration, misclassification detection, and selective prediction relative to both the strongest single-encoder baseline and naive ensemble averaging, all while preserving competitive balanced accuracy.
BibTeX
@misc{shen2026when,
  author = {Yekai Shen and Yuqian Wang and Zishun Liao and Rongsheng Wang and Wenting Chen and S. Kevin Zhou and Qingsong Yao},
  title = {{When Foundation Models Disagree: Representational Heterogeneity for Calibrated Uncertainty in Whole-Slide Image Classification}},
  year = {2026},
  eprint = {2611.26159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26159},
}
Figure