TL;DR
This paper introduces a novel framework that leverages the rich, task-agnostic representations of vision foundation models (like MedSAM2) to automatically quantify intrinsic data noise—known as aleatoric uncertainty—in medical image datasets. By analyzing the singular value energy of feature matrices, the authors design annotation-free data filtering and dynamic loss optimization strategies that consistently boost the performance and robustness of mainstream segmentation networks.
What Problem Does This Paper Address?
Medical image segmentation is a cornerstone of modern clinical workflows, aiding in everything from tumor detection to surgical planning. However, medical datasets are notoriously "noisy." Variations in imaging equipment, artifacts, and subjective or ambiguous expert annotations create high aleatoric uncertainty (inherent data randomness).
When deep learning models are trained directly on these ambiguous or mislabeled samples, they often become overconfident or learn incorrect patterns. While prior research heavily focuses on fixing model architectures or estimating epistemic uncertainty (model-side doubt), systematically measuring data-side uncertainty remains a challenge. Existing data-quantification methods typically require expensive manual difficulty ratings or rely on task-specific models that are highly prone to overfitting. This paper addresses the need for an automated, reliable, and annotation-free method to measure sample difficulty and data noise.
Methods
The core innovation lies in using a large-scale, pre-trained medical vision foundation model as a fixed feature extractor to evaluate data complexity through the lens of matrix and information theory.
- Feature Extraction & SVD: The 3D spatial feature maps of an image are extracted from the foundation model and reshaped into a 2D matrix for each class. The authors perform Singular Value Decomposition (SVD) on this matrix.
- Semantic Perception Scale ($S$): Clean, highly discriminative images ("easy samples") exhibit complex, diverse feature structures, resulting in a full-rank matrix where singular value energy decays slowly. Conversely, noisy or ambiguous images exhibit sparse, disorganized patterns, causing the singular value energy to decay rapidly (a low-rank structure). The authors compute the Shannon entropy of this energy distribution to define the "semantic perception scale."
- Aleatoric Uncertainty Value (AUV): By normalizing this scale globally, they map each sample to an AUV between 0 and 1, where scores closer to 1 indicate highly uncertain or noisy data.
Based on this metric, they introduce two application strategies:
- Aleatoric Uncertainty-Aware Data Filtering: Automatically identifying and discarding the top 5% or 10% most uncertain/noisy samples before training begins.
- Dynamic Uncertainty-Aware Optimization (DUO): A plug-and-play loss function that dynamically adjusts class weights during training based on the semantic perception scale, paired with a learnable noise estimation head to denoise labels on the fly.
Datasets
The method was comprehensively validated on five public benchmark datasets spanning both CT and MRI modalities:
- LiTS (Liver Tumor Segmentation Challenge): 3D abdominal CT scans focusing on liver and liver tumors.
- TotalSegmentator: A massive dataset containing 1,204 CT images covering 104 anatomical structures.
- WORD: Abdominal CT scans featuring 16 organs with detailed pixel-level and sparse scribble annotations.
- FeTA 2022: T2-weighted fetal brain MRI reconstructions from multiple institutions.
- KiTS23: 3D CT images for kidney, kidney tumor, and renal cyst segmentation.
Key Insights and Ideas
- Foundation Models as Objective Judges: Because vision foundation models are pre-trained on massive, diverse datasets, their latent feature spaces act as stable, task-agnostic reference frames. This allows them to judge sample difficulty objectively without overfitting to the target dataset's specific quirks.
- SVD Spectrum as a Structural Metric: The mathematical decay rate of singular values serves as an elegant proxy for image quality. Fast decay equals low structural complexity (noise/ambiguity), while slow decay signifies rich, robust features.
- Annotation-Free Flexibility: The AUV can be computed using raw images (reflecting anatomical complexity) or normalized images (reflecting annotation quality), meaning it can function purely unsupervised or leverage ground-truth masks when available.
Key Results
The authors evaluated their framework against standard baselines across CNN (nnU-Net), Transformer (Swin-UNETR), and Mamba (U-Mamba) architectures.
- Data Filtering Performance: Removing the most uncertain samples scored by MedSAM2 significantly boosted baseline performance. On the LiTS dataset, filtering out the top 10% most uncertain samples improved the baseline mean Dice score across all tasks from 75.10% to 76.45%.
- DUO Stability: Implementing the dynamic optimization strategy yielded stable, widespread improvements across all backbone networks without adding any computational overhead during inference. For instance, nnU-Net's mean Dice score rose from 75.10% to 75.69%, and Swin-UNETR improved from 73.09% to 73.64%.
Limitations and Future Work
While the framework is highly effective, its success is tightly coupled with the choice of the underlying foundation model. The experiments revealed that not all foundation models yield positive results; for example, CLIP-driven models occasionally degraded performance compared to medical-specific models like MedSAM2 and SegVol.
The authors suggest that future work should focus on extending this aleatoric uncertainty framework to weakly supervised learning, federated learning, and real-world clinical data quality management systems.
Summary updated