MetaMorph: Learning Metamorphic Image Transformation with Appearance Changes

Jian Wang, Jiarui Xing, Jason Druzgal, William M. Wells Iii, Miaomiao Zhang

TL;DR

MetaMorph is a novel deep learning framework that solves a major headache in medical imaging: registering images where anatomy has physically changed (e.g., due to a growing brain tumor). By jointly learning to segment abnormal regions and calculate geometric warps, the model uses a clever "appearance-aware" regularization to ensure that tumors don't distort the alignment of healthy tissue.

What Problem Does This Paper Address?

Deformable image registration is the process of overlaying two images—like a patient’s pre-op MRI and a post-op scan—so they match up perfectly. Most state-of-the-art algorithms assume that the "topology" of the images is the same; they expect every structure in Image A to have a direct counterpart in Image B.

However, in clinical reality, this assumption often fails. Pathology like tumors, scars, or surgical resections introduce "appearance changes"—areas where data is missing or entirely different. Standard registration models get confused by these regions, often trying to "stretch" healthy brain tissue to fill a tumor void, leading to anatomically nonsensical results. Current solutions either require slow manual masking by experts or struggle to distinguish between actual movement (geometry) and new growth (appearance).

Methods

The authors introduce MetaMorph, a predictive model that treats registration and segmentation as a single, mutually beneficial task.

Key Innovations:

  1. Joint Learning Framework: The architecture consists of two main parts: a segmentation network (to find the tumors) and a registration network (to align the images).
  2. Appearance-Aware Regularization: This is the core mathematical contribution. The authors developed a "piecewise regularization" on the tangent space of the transformations. Essentially, the model uses the predicted tumor mask to "tell" the registration algorithm which areas to ignore when calculating the smooth geometric warp. This prevents the tumor's intensity changes from "polluting" the geometric transformation of the rest of the brain.
  3. Diffeomorphic Mapping: The model uses "geodesic shooting" (based on the LDDMM framework). This ensures the resulting transformations are diffeomorphisms—meaning they are smooth, reversible, and preserve the structural integrity of the anatomy.
  4. Self-Augmenting Loop: The registration process actually helps the segmentation. By warping existing labels onto new images, the model effectively creates its own augmented training data, improving segmentation accuracy without needing thousands of manual labels.

Datasets

The model was validated using:

  • BraTS 2021: 100 public 3D T1-weighted brain MRI scans from the Brain Tumor Segmentation challenge.
  • Expert Landmarks: To verify registration accuracy, clinicians annotated 28 landmarks (such as the ventricles and corpus callosum) on 60 image pairs to measure how well the structures actually aligned.

Key Insights and Ideas

  • The "Aha" Moment: Registration and segmentation shouldn't be sequential; they should be simultaneous. A good registration needs to know where the tumors are to avoid warping them incorrectly, and a good segmentation benefits from the geometric context provided by registration.
  • Masking vs. Modeling: Unlike previous methods that simply "black out" tumors, MetaMorph uses the masks to constrain the velocity fields of the transformation. This allows for a more nuanced, piecewise-smooth alignment.
  • Efficiency: Because it is a predictive model (learning-based), once trained, it can perform these complex metamorphic transformations much faster than traditional optimization-based methods, making it a candidate for real-time surgical navigation.

Key Results

MetaMorph significantly outperformed standard baselines like VoxelMorph (VM) and Metamorphic Autoencoders (MAE):

  • Registration Accuracy: The landmark error (L2 distance) was slashed. For the brain ventricles, MetaMorph achieved an error of 4.64mm, compared to 15.02mm for VoxelMorph and 10.53mm for MAE.
  • Segmentation Performance: By using the joint framework, the segmentation accuracy (Dice score) improved across all tested backbones. For example, using a Transformer-based (UnetR) backbone, the Dice score rose from 0.861 to 0.874.
  • Qualitative Quality: Visual results show that MetaMorph produces clean, smooth transformation grids, whereas other models produced "messy" warps and artifacts around the tumor boundaries.

Limitations and Future Work

While MetaMorph is a leap forward, the authors note a few areas for growth:

  • Uncertainty Quantification: The current model provides a "best guess" for the registration. Future versions could benefit from a probabilistic approach to tell surgeons how confident the model is about the alignment near the tumor edges.
  • Detection Sensitivity: The model relies on the segmentation network's ability to identify appearance changes. If a change is too subtle to segment, the registration might still struggle.
  • Broader Applications: The authors suggest extending this to even more difficult scenarios, such as real-time automated registration for ultrasound images, where noise and appearance changes are even more prevalent.

Summary updated

Abstract

This paper presents a novel predictive model, MetaMorph, for metamorphic registration of images with appearance changes (i.e., caused by brain tumors). In contrast to previous learning-based registration methods that have little or no control over appearance-changes, our model introduces a new regularization that can effectively suppress the negative effects of appearance changing areas. In particular, we develop a piecewise regularization on the tangent space of diffeomorphic transformations (also known as initial velocity fields) via learned segmentation maps of abnormal regions. The geometric transformation and appearance changes are treated as joint tasks that are mutually beneficial. Our model MetaMorph is more robust and accurate when searching for an optimal registration solution under the guidance of segmentation, which in turn improves the segmentation performance by providing appropriately augmented training labels. We validate MetaMorph on real 3D human brain tumor magnetic resonance imaging (MRI) scans. Experimental results show that our model outperforms the state-of-the-art learning-based registration models. The proposed MetaMorph has great potential in various image-guided clinical interventions, e.g., real-time image-guided navigation systems for tumor removal surgery.
BibTeX
@inproceedings{wang2023metamorph,
  author = {Jian Wang and Jiarui Xing and Jason Druzgal and William M. Wells III and Miaomiao Zhang},
  title = {{MetaMorph: Learning Metamorphic Image Transformation with Appearance Changes}},
  booktitle = {IPMI},
  year = {2023},
  url = {https://openreview.net/forum?id=6eaNhEBuX7},
}

Loading related content...

Figure