Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Ruiyang Li, Fang Liu, Licheng Jiao, Xinglin Xie, Jiayao Hao, Shuo Li, Xu Liu, Jingyi Yang, Lingling Li, Puhua Chen, Wenping Ma

TL;DR

This paper introduces a novel framework that leverages the rich, task-agnostic representations of vision foundation models (like MedSAM2) to automatically quantify intrinsic data noise—known as aleatoric uncertainty—in medical image datasets. By analyzing the singular value energy of feature matrices, the authors design annotation-free data filtering and dynamic loss optimization strategies that consistently boost the performance and robustness of mainstream segmentation networks.

What Problem Does This Paper Address?

Medical image segmentation is a cornerstone of modern clinical workflows, aiding in everything from tumor detection to surgical planning. However, medical datasets are notoriously "noisy." Variations in imaging equipment, artifacts, and subjective or ambiguous expert annotations create high aleatoric uncertainty (inherent data randomness).

When deep learning models are trained directly on these ambiguous or mislabeled samples, they often become overconfident or learn incorrect patterns. While prior research heavily focuses on fixing model architectures or estimating epistemic uncertainty (model-side doubt), systematically measuring data-side uncertainty remains a challenge. Existing data-quantification methods typically require expensive manual difficulty ratings or rely on task-specific models that are highly prone to overfitting. This paper addresses the need for an automated, reliable, and annotation-free method to measure sample difficulty and data noise.

Methods

The core innovation lies in using a large-scale, pre-trained medical vision foundation model as a fixed feature extractor to evaluate data complexity through the lens of matrix and information theory.

  1. Feature Extraction & SVD: The 3D spatial feature maps of an image are extracted from the foundation model and reshaped into a 2D matrix for each class. The authors perform Singular Value Decomposition (SVD) on this matrix.
  2. Semantic Perception Scale ($S$): Clean, highly discriminative images ("easy samples") exhibit complex, diverse feature structures, resulting in a full-rank matrix where singular value energy decays slowly. Conversely, noisy or ambiguous images exhibit sparse, disorganized patterns, causing the singular value energy to decay rapidly (a low-rank structure). The authors compute the Shannon entropy of this energy distribution to define the "semantic perception scale."
  3. Aleatoric Uncertainty Value (AUV): By normalizing this scale globally, they map each sample to an AUV between 0 and 1, where scores closer to 1 indicate highly uncertain or noisy data.

Based on this metric, they introduce two application strategies:

  • Aleatoric Uncertainty-Aware Data Filtering: Automatically identifying and discarding the top 5% or 10% most uncertain/noisy samples before training begins.
  • Dynamic Uncertainty-Aware Optimization (DUO): A plug-and-play loss function that dynamically adjusts class weights during training based on the semantic perception scale, paired with a learnable noise estimation head to denoise labels on the fly.
Illustrative pipeline of the proposed uncertainty quantification and application framework.

Datasets

The method was comprehensively validated on five public benchmark datasets spanning both CT and MRI modalities:

  • LiTS (Liver Tumor Segmentation Challenge): 3D abdominal CT scans focusing on liver and liver tumors.
  • TotalSegmentator: A massive dataset containing 1,204 CT images covering 104 anatomical structures.
  • WORD: Abdominal CT scans featuring 16 organs with detailed pixel-level and sparse scribble annotations.
  • FeTA 2022: T2-weighted fetal brain MRI reconstructions from multiple institutions.
  • KiTS23: 3D CT images for kidney, kidney tumor, and renal cyst segmentation.

Key Insights and Ideas

  • Foundation Models as Objective Judges: Because vision foundation models are pre-trained on massive, diverse datasets, their latent feature spaces act as stable, task-agnostic reference frames. This allows them to judge sample difficulty objectively without overfitting to the target dataset's specific quirks.
  • SVD Spectrum as a Structural Metric: The mathematical decay rate of singular values serves as an elegant proxy for image quality. Fast decay equals low structural complexity (noise/ambiguity), while slow decay signifies rich, robust features.
  • Annotation-Free Flexibility: The AUV can be computed using raw images (reflecting anatomical complexity) or normalized images (reflecting annotation quality), meaning it can function purely unsupervised or leverage ground-truth masks when available.

Key Results

The authors evaluated their framework against standard baselines across CNN (nnU-Net), Transformer (Swin-UNETR), and Mamba (U-Mamba) architectures.

  • Data Filtering Performance: Removing the most uncertain samples scored by MedSAM2 significantly boosted baseline performance. On the LiTS dataset, filtering out the top 10% most uncertain samples improved the baseline mean Dice score across all tasks from 75.10% to 76.45%.
  • DUO Stability: Implementing the dynamic optimization strategy yielded stable, widespread improvements across all backbone networks without adding any computational overhead during inference. For instance, nnU-Net's mean Dice score rose from 75.10% to 75.69%, and Swin-UNETR improved from 73.09% to 73.64%.
Comparison of segmentation boundaries showing how the DUO strategy (orange) aligns closer to the ground truth (red) than the baseline (blue).

Limitations and Future Work

While the framework is highly effective, its success is tightly coupled with the choice of the underlying foundation model. The experiments revealed that not all foundation models yield positive results; for example, CLIP-driven models occasionally degraded performance compared to medical-specific models like MedSAM2 and SegVol.

The authors suggest that future work should focus on extending this aleatoric uncertainty framework to weakly supervised learning, federated learning, and real-world clinical data quality management systems.

Summary updated

Abstract

Medical image segmentation supports clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets medical image datasets suffer from acquisition noise and annotation ambiguity, causing pervasive data uncertainty that substantially undermines model robustness. Existing research focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures, revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks.
BibTeX
@inproceedings{li2026delving,
  author = {Ruiyang Li and Fang Liu and Licheng Jiao and Xinglin Xie and Jiayao Hao and Shuo Li and Xu Liu and Jingyi Yang and Lingling Li and Puhua Chen and Wenping Ma},
  title = {{Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=lhTDFnGxTq},
}
Figure