Overcoming the Modality Gap in Medical Image Segmentation

341 concepts274 papers10 proposals

Deep learning models excel at segmenting organs or tumors when the test data perfectly matches the training data. However, if you switch from a Computed Tomography (CT) scan to a Magnetic Resonance Image (MRI), or even between different MRI pulse sequences (like T1-weighted to T2-weighted), model performance typically collapses.

This failure occurs because neural networks are highly sensitive to modality-specific shortcuts. Instead of learning the invariant geometric and topological relationships of human anatomy, they memorize low-level sensor characteristics: pixel intensity distributions, local textures, and hardware-specific noise patterns. For instance, a model might learn that a liver corresponds to a specific range of Hounsfield Units in a CT scan. When presented with an MRI where those intensities are entirely different, the model's internal representations break down.

To build truly generalizable medical AI, researchers must force networks to ignore these superficial stylistic variations and focus entirely on underlying anatomical structures. Three main paradigms have emerged to achieve this: domain randomization, frequency-domain disentanglement, and synergistic feature alignment.

Forcing Geometric Invariance via Domain Randomization

If a network fails because it relies too heavily on pixel intensities, one intuitive solution is to make pixel intensities completely uninformative. By training a model on data where contrasts and textures change radically from image to image, the network is forced to abandon intensity shortcuts and instead learn shape, boundaries, and spatial context.

This strategy is powerfully demonstrated by SynthSeg, an architecture introduced by Douglas N. Greve et al. in A Learning Strategy for Contrast-agnostic MRI Segmentation. Instead of training on real, meticulously labeled MRIs, the authors trained a network entirely on synthetic images generated dynamically from anatomical label maps. During training, the tissue intensities, spatial noise, and artifacts of these synthetic images are completely randomized. Because the network never sees a stable intensity profile, it cannot memorize modality-specific shortcuts. Consequently, the resulting model can segment brain structures across entirely unseen, real MRI sequences with high fidelity.

[Anatomical Label Map] ──> [Randomize Intensities & Noise] ──> [Synthetic Image] ──> [Train Segmentation Model]
                                                                                              │
                                                                     (Forces model to learn shape, not contrast)

Taking this philosophy to its logical extreme, Neel Dey et al. developed AnyStar in AnyStar: Domain randomized universal star-convex 3D instance segmentation. Rather than simulating specific human anatomy, the authors trained a 3D instance segmentation network exclusively on synthetic, abstract "blob-like" shapes overlaid with extreme variations in signal-to-noise ratios and contrast artifacts. Because the model learns to rely purely on geometric properties like star-convexity, a single trained model successfully generalizes across five distinct real-world imaging modalities—ranging from fluorescence microscopy to CT—without ever seeing a single real image during training.

Disentangling Style and Structure in the Frequency Domain

Another elegant way to bypass the modality gap is to separate an image's style from its structure using the Fourier Transform. In medical imaging, an image can be decomposed into two primary components:

  1. The Amplitude Spectrum: This captures low-frequency global information, which corresponds to the visual "style," contrast, and specific modality characteristics.
  2. The Phase Spectrum: This preserves high-frequency structural information, which corresponds to anatomical boundaries, shapes, and object contours.

Because the phase spectrum remains relatively invariant across modalities while the amplitude spectrum fluctuates wildly, manipulating these components allows models to normalize out the modality signature.

                       ┌──> Amplitude Spectrum (Style/Contrast) ──> Normalize/Swap
[Medical Image] ──FT───┤
                       └──> Phase Spectrum (Anatomy/Boundaries) ───> Preserve Intact

In FAN-Net: Fourier-Based Adaptive Normalization For Cross-Domain Stroke Lesion Segmentation, Weiyi Yu et al. leveraged this property to segment stroke lesions across different clinical sites and MRI sequences. By adaptively normalizing only the low-frequency amplitude components of the target data while strictly preserving the phase components, the model strips away site-specific appearance variations without altering the underlying pathology boundaries.

Similarly, Yifan Bo et al. applied this insight to few-shot learning in FAMNet: Frequency-aware Matching Network for Cross-domain Few-shot Medical Image Segmentation. Standard few-shot networks struggle to match a support image to a query image if they come from different modalities, because spatial feature matching is easily fooled by intensity inversions. By shifting the feature-matching process from the volatile spatial domain to the frequency domain, the authors bridged the appearance gap between modalities like CT and MRI, allowing successful segmentation with minimal target-domain labels.

Synergistic Image and Feature Alignment

When paired datasets (e.g., identical patients scanned in both CT and MRI) are unavailable, Unsupervised Domain Adaptation (UDA) frameworks attempt to align a labeled source modality with an unlabeled target modality. However, aligning only the raw pixel appearance or only the latent features often leaves a residual domain gap. The most robust architectures perform alignment at both levels simultaneously.

This dual-alignment intuition was formalized by Cheng Chen et al. through the SIFA framework in Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation. SIFA uses a generative adversarial network (GAN) to translate the visual appearance of source domain images into the target domain style (pixel-level alignment). Concurrently, it uses adversarial discriminators deeper in the network to ensure that the latent features extracted from both real target images and translated source images share the same mathematical distribution (feature-level alignment).

By unifying these two perspectives, the network ensures that its internal feature representations become entirely modality-invariant. Expanding on this symmetric philosophy, Lei Qi et al. proposed the Deep Symmetric Adaptation Network for Cross-modality Medical Image Segmentation, which utilizes a bidirectional translation architecture to align image styles in both directions, further stabilizing the shared segmentation representations.

Mapping to Invariant Causal and Simulation Spaces

Rather than trying to force two highly disparate modalities to look like one another, some approaches construct an explicit intermediate space or leverage causal graphs to isolate anatomy from environmental noise.

For example, when transitioning between modalities based on completely different physical principles—such as CT (X-ray attenuation) and Ultrasound (acoustic reflection)—the domain gap is too wide for simple style transfer. To bridge this, Yordanka Velikova developed CACTUSS in CACTUSS: Common Anatomical CT-US Space for US examinations. The author simulated an intermediate, physics-informed "Common Anatomical Space" that blends the structural properties of both modalities, allowing models trained on high-quality CT data to generalize directly to low-cost ultrasound imaging without requiring manual ultrasound annotations.

From a theoretical perspective, this separation can be framed as an invariant causal mechanism. In Generalizable Single-Source Cross-Modality Medical Image Segmentation via Invariant Causal Mechanisms, Boqi Chen et al. modeled the medical image generation process as a combination of two independent factors: a causal anatomical factor (the true shape of the organ) and an environmental style factor (the imaging modality). By using generative diffusion models to intervene on the style factor while holding the anatomical structure strictly constant, the authors simulated an endless variety of imaging modalities. This causal intervention prevents the downstream segmentation network from ever overfitting to a single sensor distribution, achieving robust single-source domain generalization.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 251 papers
  2. Version 1 · 2026-10-04 · 237 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modality_domain_shift_f1b34825,
  title = {cross-modality domain shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modality-domain-shift/rdr_f1b3482583744f148fabd2ea69891e49},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_cross_modality_domain_shift_f1b34825_v2,
  title = {cross-modality domain shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modality-domain-shift/rdr_f1b3482583744f148fabd2ea69891e49/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake