Adapting Medical Image Segmentation to Unseen Clinical Domains

20 concepts20 papers1 proposal

Deep learning models excel at segmenting organs or lesions when evaluated on the exact same dataset they were trained on. However, when deployed to a new clinical site, their performance often plummets. This "domain shift" is driven by variations in scanner manufacturers, imaging protocols, noise levels, and patient demographics.

Crucially, while the visual style (contrast, brightness, and resolution) changes across hospitals, the underlying human anatomy remains invariant. To build models that work reliably everywhere, researchers focus on a core intuition: disentangling site-specific style from universal anatomical structure.

Disentangling Style from Anatomy via the Frequency Domain

How do we force a neural network to ignore superficial textures and focus entirely on shape? One powerful approach relies on the Fourier transform. In an image's frequency spectrum, the amplitude component largely captures low-frequency style information (such as contrast and lighting), while the phase component preserves high-frequency structural boundaries (the actual anatomy).

Input Image ───► Fourier Transform ───► Amplitude (Style / Contrast) ───► Perturb / Normalize
                                   ───► Phase (Structure / Anatomy)  ───► Preserve Intact

Yu et al. leverage this property in their FAN-Net framework by adaptively normalizing low-frequency amplitude components across domains while strictly preserving phase components. This allows the network to standardize the visual appearance of MRI scans without distorting lesion boundaries. Similarly, Qiao et al. and Li et al. use frequency-mixing strategies during training. By deliberately perturbing or shuffling the amplitude spectrum while keeping the phase intact, they force the model to become invariant to style shifts before it ever encounters a new clinical site.

Beyond the frequency domain, this separation can be achieved via explicit shape modeling or image translation. For instance, Yao et al. utilize Variational Autoencoders (VAEs) to model organ shapes independently of texture, ensuring that geometric understanding remains intact even if a new scanner alters an image's appearance. Alternatively, Chen et al. (2018) use Generative Adversarial Networks (GANs) to translate target images so they visually match the source domain, employing a semantic-aware loss to guarantee that anatomical structures are not altered during translation.

Adapting on the Fly: Source-Free and Test-Time Adaptation

In many real-world clinical settings, privacy regulations prevent developers from accessing the original training data once a model is deployed. When a model encounters an unfamiliar data distribution at a new hospital, it must adapt using only the incoming, unlabeled test images—a paradigm known as Source-Free Domain Adaptation (SFDA) or Test-Time Adaptation (TTA).

Without ground-truth labels, models must generate their own guidance, often through "pseudo-labels" (treating the model's high-confidence predictions as truth). However, domain shift causes models to make confident mistakes, leading to error accumulation. To solve this, Chen et al. (2021) introduce Denoised Pseudo-Labeling (DPL), which combines pixel-level uncertainty estimation with class-level feature prototypes to filter out noisy mistakes, allowing the model to safely teach itself the characteristics of the new domain.

Instead of modifying the core weights of a fragile segmentation network during inference—which risks catastrophic forgetting—an alternative strategy is to adapt the input or use lightweight adapters:

  • Image Transformation: Tomar et al. introduce OptTTA, which learns to dynamically transform incoming test images so they "look" like the data the model was originally trained on. Similarly, Xie et al. propose a consistency-based metric to intelligently select the optimal image scale and style augmentations at test time.
  • Visual Prompting: Chen et al. (2024) keep the base model weights completely frozen and instead optimize a tiny, image-specific "visual prompt" in the frequency domain for each test scan. To make this process computationally viable for real-time clinical workflows, Liu et al. (2025) introduce an Efficient Deformable Convolutional Prompt (EDCP) that adaptively warps inputs to align with the model's learned representations.

When updates are made to model parameters during test time, ensuring stability is critical. Chen et al. (2025) address this with GraTa, a method that aligns the gradients of two distinct self-supervised tasks. By ensuring that different self-supervised objectives agree on the direction of the model updates, they prevent the model from drifting into degenerate solutions during inference.

Simulating Variance via Domain Generalization

If a model cannot be updated at test time, it must be made inherently robust during its initial training phase—a goal known as Domain Generalization (DG). When developers only have access to a single source dataset, they must artificially simulate the environmental variance the model will eventually encounter in the wild.

Xu et al. achieve this by using an adversarial synthesizer to generate "hard" training examples that push the boundaries of the source distribution, combined with patch-level contrastive learning to ensure the underlying anatomy remains consistent. Rather than manually designing these synthetic variations, Lyu et al. employ reinforcement learning to automatically discover optimal augmentation policies, maximizing the diversity of simulated domains during training to prepare the model for unseen modalities.

The Broader Impact of Distribution Shifts

The challenges of domain shift extend beyond static deployment. As Liu et al. (2022) demonstrate, when models are trained incrementally to recognize new organs over time, "catastrophic forgetting" is often driven more by underlying distribution shifts between sequential datasets (such as changing scanner protocols) than by the mere absence of labels.

As highlighted by the benchmarking work of Saat et al., no single adaptation strategy consistently outperforms all others across every clinical scenario. However, by continuously refining how models separate invariant anatomical structures from volatile scanner styles, the field moves closer to plug-and-play medical AI that performs reliably across diverse hospital networks.

Key Foundational Work

Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling (2021) — contributed 1 concept elements to this direction.
Semantic-Aware Generative Adversarial Nets for Unsupervised Domain Adaptation in Chest X-ray Segmentation (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_domain_shift_in_medical_imaging_domain_s_2f7aa111,
  title = {domain shift in medical imaging (domain shift)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-shift-in-medical-imaging-domain-shift/rdr_2f7aa111ca5c413f94a6fff756eee11e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_domain_shift_in_medical_imaging_domain_s_2f7aa111_v1,
  title = {domain shift in medical imaging (domain shift)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-shift-in-medical-imaging-domain-shift/rdr_2f7aa111ca5c413f94a6fff756eee11e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake