Generalizing Medical Image Analysis Across Modalities and Anatomies

70 concepts71 papers1 proposal

Deep learning models in medical imaging have traditionally been "specialists," trained to perform a single task on a specific imaging modality, such as segmenting the liver in CT scans or identifying tumors in T2-weighted MRI. However, this specificity creates brittleness: a model trained on one scanner often fails when deployed on another due to variations in signal-to-noise ratios, voxel resolutions, or tissue contrast.

Recent research has shifted toward cross-modality generalization, aiming to build "generalist" models that maintain performance across diverse anatomical structures and imaging protocols. This direction moves away from task-specific engineering toward architectures that capture invariant features of human anatomy and fundamental visual cues.

The Shift Toward Unified Architectures

The most direct approach to generalization is the development of unified architectures capable of handling heterogeneous data. Rather than designing a new network for every organ, researchers are building models that treat diverse medical imaging tasks as facets of a single problem.

The Medical Segmentation Decathlon (MSD), introduced by Simpson et al., formalized this by providing a benchmark of ten diverse datasets. The authors argue that multi-task benchmarks drive the development of "AutoML" algorithms that generalize across organs and modalities without manual per-task tuning. Similarly, Cheng et al. proposed CE-Net, which uses a unified context-encoding architecture to achieve state-of-the-art results across CT, fundus imaging, and electron microscopy. By prioritizing multi-scale context extraction, these models become robust to the varying visual characteristics of different datasets.

Decoupling Anatomy from Imaging Physics

A key intuition behind cross-modality success is that while the appearance of an organ changes between a CT and an MRI, its underlying shape does not. By decoupling anatomical priors from image intensity, models can learn to recognize structures regardless of the imaging physics.

Dalca demonstrated this by using anatomical priors learned from unpaired labels. This allows a model to segment scans in a new modality where no labeled data exists, simply by relying on the consistency of human anatomy. This principle of "mask transferability" is further explored by Wang et al., who showed that segmentation masks derived from MRI data can effectively supervise the training of CT models. Because the anatomical structures are shared across imaging physics, the labels themselves become a bridge between domains.

Modular Adaptation and Visual Prompting

As models grow in complexity, retraining them for every new clinical context becomes computationally prohibitive. A newer strategy involves keeping a powerful, pre-trained "foundation" model frozen and adapting it using small, modular components.

The Medical Visual Prompting (MVP) framework, proposed by Huang et al., adapts a frozen vision transformer to CT, MRI, and endoscopy using "visual prompts." By injecting superpixel-based shape information into the model, the authors achieve high-quality segmentation across modalities while training only a tiny fraction of the total parameters. This suggests that the high-level features learned by large-scale models are sufficiently general to be "steered" toward specific medical tasks with minimal effort.

Learning Invariant Visual Cues

Beyond shape, certain low-level visual features—like edges, textures, and pixel correspondences—remain consistent across modalities. Research into Instructive Feature Enhancement (IFE) by Liu et al. shows that adaptively boosting features rich in texture and curvature information enables universal performance across seven different imaging modalities.

Similarly, Yeung et al. developed Sli2Vol, which prioritizes pixel correspondence between adjacent slices rather than object classification. By focusing on how tissues move and relate to one another in 3D space, the model remains agnostic to specific anatomical structures, allowing a system trained on abdominal data to successfully segment the heart or lungs without retraining.

Robustness in Clinical Practice

The ultimate goal of this research is clinical reliability. In real-world settings, imaging sequences are often missing or acquired on different hardware. Qiu et al. addressed this with MyoPS-Net, a flexible architecture that maintains accuracy even when specific MRI sequences are unavailable. By training on heterogeneous datasets, the authors found that a single, flexible model generalizes better across diverse hospital environments than specialized models.

Whether through hierarchical transformers like UNesT (Yu et al.) or diffusion-based models like SpineSegDiff (Monzon et al.), the field is converging on a central truth: the most robust medical AI is not the one that learns a single task perfectly, but the one that learns the underlying language of medical imaging—a language spoken across all modalities and anatomies.

Key Foundational Work

Segmentation of Left Ventricle via Level Set Method Based on Enriched Speed Term (2004) — contributed 1 concept elements to this direction.
A General System for Automatic Biomedical Image Segmentation Using Intensity Neighborhoods (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modality_generalization_cross_moda_737f5013,
  title = {cross-modality generalization (cross-modality generalizability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modality-generalization-cross-modality-generalizability/rdr_737f5013fbcd4e8f8e82598812b6322b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modality_generalization_cross_moda_737f5013_v1,
  title = {cross-modality generalization (cross-modality generalizability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modality-generalization-cross-modality-generalizability/rdr_737f5013fbcd4e8f8e82598812b6322b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake