Joint Learning of Registration and Anatomical Segmentation

This direction is split into other directions: alignment sensitivity (spatial registration dependency). Its last version stays readable and citable here.

Version 1 of 1150 papers170 conceptsfrozen 2026-10-04

In medical imaging, the tasks of identifying "what" a structure is (segmentation) and "where" it is relative to a standard map (registration) are traditionally treated as separate, sequential steps. However, a growing body of research suggests that these two problems are deeply interdependent. By treating them as a single, joint optimization problem, researchers are overcoming the "chicken-and-egg" cycle where poor alignment ruins segmentation and inaccurate segmentation prevents precise alignment.

The Feedback Loop of Anatomical Priors

The core intuition behind this research direction is that anatomical knowledge should act as a "topological roadmap." When we segment a brain or a liver, we aren't just looking at pixel intensities; we are looking for structures that we know exist in a specific spatial relationship to one another.

Udaranga Wickramasinghe and colleagues demonstrated this with PA-Net (Probabilistic Atlases to Enforce Topological Constraints), which uses deformable atlases to prevent neural networks from making anatomically impossible mistakes. Similarly, Xiang Chen introduced textSCF (Spatially Covariant Image Registration with Text Prompts), which uses vision-language models to apply different registration "logic" to different organs based on their names.

In these systems, registration provides the spatial prior—the expectation of where an organ should be—while segmentation provides the semantic evidence to refine that expectation.

The Problem of Error Propagation

The danger of the traditional sequential approach is that errors in the first step compound in the second. If a registration algorithm fails to account for the "sliding" motion between abdominal organs, as noted by Seyoun Park (Multi-phase Deformable Registration for Time-dependent Abdominal Organ Variations), any subsequent analysis of those organs will be spatially misaligned.

This "propagation of error" is a recurring theme across the literature:

Closing the Loop: Joint Optimization

To break this cycle, researchers are moving toward frameworks that learn registration and segmentation simultaneously. The intuition is that if the model knows what the structure is, it can align it better; and if it aligns it better, it can identify it more accurately.

Jian Li and Adrian V Dalca demonstrated this with JParc (Joint cortical surface parcellation with registration), showing that explicit anatomical registration is often the "secret sauce" that provides the majority of predictive accuracy in brain mapping. Similarly, Benoît Scherrer used interlocking Markov Random Fields to allow tissue and structure data to inform one another in real-time (Fully Bayesian Joint Model for MR Brain Scan Tissue and Structure Segmentation).

In the cardiac domain, Xiang Chen developed SDDIR (Joint segmentation and discontinuity-preserving deformable registration), which allows for realistic "sliding" motions at tissue boundaries by jointly learning to segment the organs and preserve motion discontinuities.

Beyond Segmentation: Representation and Reconstruction

This joint approach is extending into broader areas of medical AI. Li Sun and Ke Yu have shown that self-supervised representation learning is more effective when it is "anatomy-conditioned" through registration (Context Matters; DrasCLR). By forcing a model to understand its location within a standard atlas, it learns to ignore "easy" anatomical landmarks and focus on subtle, disease-related features.

Even the fundamental process of image reconstruction is being reimagined. Angelica I Aviles-Rivero and Hui Ji introduced a framework that simultaneously reconstructs an image from raw sensor data while registering it to a template (Deep Joint Indirect Registration and Reconstruction). This uses the "shape prior" of the atlas to fill in missing information during the reconstruction process itself.

The shift toward joint modeling reflects a maturing field: moving away from isolated algorithmic tasks and toward a holistic, biologically-grounded understanding of medical images.

Papers in this version

Showing 60 of 150 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_learning_of_registration_and_anato_fd9dba47,
  title = {Joint Learning of Registration and Anatomical Segmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-learning-of-registration-and-anatomical-segmentation/rdr_fd9dba47d02c4419946a8b08c7125c52},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_learning_of_registration_and_anato_fd9dba47_v1,
  title = {Joint Learning of Registration and Anatomical Segmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-learning-of-registration-and-anatomical-segmentation/rdr_fd9dba47d02c4419946a8b08c7125c52/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}