SUD: Supervision by Denoising for Medical Image Segmentation

Sean I. Young, Adrian V. Dalca, Enzo Ferrante, Polina Golland, Bruce Fischl, Juan Eugenio Iglesias

Abstract

Deep learning models for medical image segmentation often struggle due to the scarcity and variability of expert-labeled data. We present Supervision by Denoising (SUD), a semi-supervised framework that leverages unlabeled data by using the model's own denoised predictions as pseudo-labels. By combining spatial denoising with temporal averaging, SUD provides a stable anatomical prior that significantly improves performance in low-data regimes, such as "one-shot" learning scenarios.

Introduction

Standard architectures like the U-Net are highly effective but data-hungry. In medical imaging, obtaining pixel-level labels is expensive and prone to inter-rater variability. While semi-supervised learning (SSL) offers a path forward by utilizing unlabeled data, traditional SSL methods often rely on handcrafted regularizers that fail to capture complex anatomical structures. SUD addresses this by using a learned denoiser to act as an anatomical prior, supervising the training process on unlabeled images.

The SUD Framework

The core of SUD is a loop that alternates between prediction and denoising to generate high-quality soft targets for unlabeled data.

The SUD framework architecture, illustrating the integration of spatial denoising and temporal averaging to generate pseudo-labels.

The process involves three key steps:

  1. Prediction: A "student" network predicts a segmentation map for an unlabeled input.
  2. Spatio-Temporal Denoising: The prediction is refined using a pre-trained spatial denoiser (an auto-encoder) to enforce anatomical plausibility and a temporal filter to average predictions over training steps, reducing stochastic noise.
  3. Supervision: The student network is updated to minimize the loss between its raw prediction and this refined "soft label."

The authors found that simple auto-encoders, which lack skip connections, are superior denoisers because they must learn a compressed global representation of anatomy, effectively "inpainting" noisy or missing regions.

Experimental Validation

SUD was evaluated on two primary tasks:

  • 2D Cortical Parcellation: Mapping cerebral cortex folds using Mindboggle and HCP datasets.
  • 3D Anatomical Brain Reconstruction: Segmenting 3D MRIs across multiple large-scale datasets (e.g., UK Biobank, ABIDE).

Denoisers were pre-trained on label maps generated by FreeSurfer, allowing the system to learn anatomical priors independently of the primary segmentation task.

Key Results

SUD demonstrates substantial improvements over supervised-only baselines and existing stochastic averaging methods.

Qualitative and quantitative comparison of cortical parcellation. SUD (c) achieves a Dice score of 0.8206, significantly outperforming the standard U-Net (b) and closely matching the reference (d).
  • Low-Data Performance: In "one-shot" scenarios (only one labeled image), SUD improved Dice scores by approximately 10 points over a standard U-Net and 5 points over ensembling methods.
  • Boundary Accuracy: The 95% Hausdorff Distance was reduced by 50–75%, indicating much higher precision at anatomical boundaries.
  • Robustness: SUD-trained models generalized better across different scanners and hospital sites, mitigating the common issue of distribution shift in medical AI.

Conclusion

Supervision by Denoising offers a powerful, learned alternative to handcrafted regularizers in semi-supervised medical imaging. By treating a denoiser as an anatomical prior, SUD produces segmentations that are not only more accurate but also more anatomically plausible. Future work will explore combining SUD with advanced data augmentation to further reduce the reliance on human-annotated data.

Summary updated

Abstract

Learning-based image reconstruction models, such as those based on the U-Net, require a large set of labeled images if good generalization is to be guaranteed. In some imaging domains, however, labeled data with pixel- or voxel-level label accuracy are scarce due to the cost of acquiring them. This problem is exacerbated further in domains like medical imaging, where there is no single ground truth label, resulting in large amounts of repeat variability in the labels. Therefore, training reconstruction networks to generalize better by learning from both labeled and unlabeled examples (called semi-supervised learning) is problem of practical and theoretical interest. However, traditional semi-supervised learning methods for image reconstruction often necessitate handcrafting a differentiable regularizer specific to some given imaging problem, which can be extremely time-consuming. In this work, we propose "supervision by denoising" (SUD), a framework that enables us to supervise reconstruction models using their own denoised output as soft labels. SUD unifies stochastic averaging and spatial denoising techniques under a spatio-temporal denoising framework and alternates denoising and model weight update steps in an optimization framework for semi-supervision. As example applications, we apply SUD to two problems arising from biomedical imaging -- anatomical brain reconstruction (3D) and cortical parcellation (2D) -- to demonstrate a significant improvement in the image reconstructions over supervised-only and stochastic averaging baselines.
BibTeX
@article{young2025sud,
  author = {Sean I. Young and Adrian V. Dalca and Enzo Ferrante and Polina Golland and Bruce Fischl and Juan Eugenio Iglesias},
  title = {{SUD: Supervision by Denoising for Medical Image Segmentation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2023.3299789},
}
Figure