Multi-scale Neural ODEs for 3D Medical Image Registration

Junshen Xu, Eric Z. Chen, Xiao Chen, Terrence Chen, Shanhui Sun

TL;DR

This paper introduces MS-ODENet, a framework that treats 3D medical image registration as a continuous optimization process governed by Neural Ordinary Differential Equations (ODEs). By combining a multi-scale architecture with a modality-independent similarity metric, the authors achieve state-of-the-art registration accuracy across different MRI contrasts (e.g., T1 to T2) while being significantly faster than traditional iterative methods.

What Problem Does This Paper Address?

Image registration—the process of aligning two or more images into a single coordinate system—is a cornerstone of medical AI, used for everything from tracking tumor growth to surgical planning.

The field currently faces a "speed vs. accuracy" trade-off:

  1. Conventional Methods (like ANTs): These use iterative optimization. They are highly accurate but painfully slow, often taking minutes or hours per volume, which hinders real-time clinical use.
  2. Deep Learning "Learn-to-Map" Methods (like VoxelMorph): These use a single neural network pass to predict alignment. They are incredibly fast but often struggle with large anatomical movements or "multi-modal" images (where the same organ looks different because it was captured using different MRI settings).

The authors aim to bridge this gap by learning a "registration optimizer" that is as accurate as iterative methods but as efficient as deep learning.

Methods

The researchers propose two major innovations:

  • Multi-scale Neural ODEs (MS-ODENet): Instead of a standard "black box" network, they model the registration as a continuous trajectory. The network learns the derivative (the rate of change) of the transformation parameters. To handle large motions efficiently, they solve this ODE across an image pyramid (coarse-to-fine). This allows the model to fix big misalignments at low resolutions before refining the details at high resolutions.
  • Modality-Independent Similarity Metric: To register images of different types (e.g., a T1-weighted MRI to a T2-weighted MRI), the model needs to know what "success" looks like. The authors used a Generative Adversarial Network (GAN) to "disentangle" images into content (anatomy) and style (the MRI contrast). By comparing only the "content" features during registration, the model ignores the distracting visual differences between MRI types.

The training is unsupervised, meaning it doesn't require "ground truth" alignments (which are rare and expensive to produce). Instead, it uses a self-supervised loss where the model learns to undo random perturbations it applies to the data.

Datasets

The study utilized two primary data sources:

  1. BraTS 2020 (Public): A dataset of 494 subjects with glioblastomas (brain tumors). It includes four MRI contrasts (T1, T2, T2-FLAIR, T1Gd), providing 12 possible multi-modal registration pairs per subject.
  2. Private Volunteer Dataset: 3D MR brain data from 25 volunteers, used specifically to test how well the model generalizes to new, unseen data sources.

Key Insights and Ideas

  • Neural ODEs as "Learned Optimizers": Traditional optimizers only look at the local gradient of the current image pair. MS-ODENet, however, "remembers" patterns from thousands of training examples, allowing it to navigate the optimization landscape more effectively and avoid getting stuck in "local minima" (false alignments).
  • Memory Efficiency: By using the "adjoint method" to train the Neural ODE, the model doesn't need to store every intermediate step in memory. This makes it much more memory-efficient than "cascaded" deep learning models that simply stack multiple networks.
  • The Power of Disentanglement: By pre-training a feature extractor to ignore "style," the registration network can treat multi-modal registration (T1 to T2) as if it were mono-modal (T1 to T1), greatly simplifying the task.

Key Results

The MS-ODENet outperformed both classical and deep learning baselines across rigid, deformable, and hybrid registration tasks:

  • Accuracy: On the BraTS dataset for rigid registration, MS-ODENet achieved a 90.6% Dice score (a measure of overlap), compared to just 63.0% for the industry-standard ANTs.
  • Speed: MS-ODENet completed rigid registrations in 0.55 seconds, while ANTs took over 17 seconds. For complex "rigid + deformable" tasks, MS-ODENet remained under 1 second, while ANTs took over 70 seconds.
  • Robustness: The multi-scale approach proved essential for large motions. Without the multi-scale ODE, accuracy dropped significantly as the number of solver steps increased.

Limitations and Future Work

  • GAN Training Complexity: Training the 2D multi-channel GAN to extract content features is a separate, computationally intensive step. The authors noted that training this in full 3D is currently difficult due to high dimensionality.
  • Specific Anatomies: While highly successful on brain MRIs, the model's performance on highly deformable organs (like the lungs or liver) remains to be tested.
  • Future Directions: The authors plan to extend this framework to 3D-2D registration (e.g., aligning a 3D CT scan to a 2D X-ray), which is a critical challenge in image-guided surgery.

Summary updated

Abstract

Image registration plays an important role in medical image analysis. Conventional optimization based methods provide an accurate estimation due to the iterative process at the cost of expensive computation. Deep learning methods such as learn-to-map are much faster but either iterative or coarse-to-fine approach is required to improve accuracy for handling large motions. In this work, we proposed to learn a registration optimizer via a multi-scale neural ODE model. The inference consists of iterative gradient updates similar to a conventional gradient descent optimizer but in a much faster way, because the neural ODE learns from the training data to adapt the gradient efficiently at each iteration. Furthermore, we proposed to learn a modal-independent similarity metric to address image appearance variations across different image contrasts. We performed evaluations through extensive experiments in the context of multi-contrast 3D MR images from both public and private data sources and demonstrate the superior performance of our proposed methods.
BibTeX
@inproceedings{xu2021multi,
  author = {Junshen Xu and Eric Z. Chen and Xiao Chen and Terrence Chen and Shanhui Sun},
  title = {{Multi-scale Neural ODEs for 3D Medical Image Registration}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=QUJTHKCDVA8},
}

Loading related content...

Figure