Medical Image Registration via Neural Fields

Shanlin Sun, Kun Han, Chenyu You, Hao Tang, Deying Kong, Junayed Naushad, Xiangyi Yan, Haoyu Ma, Pooya Khosravi, James S. Duncan, Xiaohui Xie

TL;DR

The paper introduces Neural Image Registration (NIR), a novel framework that uses coordinate-based neural fields to perform medical image registration via pair-wise optimization. By representing deformations as continuous functions parameterized by lightweight MLPs, NIR achieves state-of-the-art alignment accuracy and topology preservation while running significantly faster and using less memory than traditional optimization methods.

What Problem Does This Paper Address?

Medical image registration—the process of aligning two or more 3D medical scans (like MRIs) into a common coordinate space—is foundational for tasks like disease tracking and atlas-based segmentation.

Historically, this has been solved using optimization-based methods (e.g., SyN, NiftyReg) that iteratively find the best transformation for a specific pair of images. While highly accurate and reliable, they are computationally slow and rely on rigid, hand-crafted modeling assumptions. Conversely, recent learning-based methods (e.g., VoxelMorph) train deep convolutional networks to predict transformations instantly. However, these models often generalize poorly to unseen image domains or unique patient anatomy not represented in the training data.

The authors bridge this gap by asking: Can we leverage the expressive power of deep neural networks within an optimization-based framework?

Methods

NIR reformulates registration by modeling the transformation between two images as a continuous function using Neural Fields (also known as Implicit Neural Representations). Instead of predicting a dense deformation grid all at once, a lightweight Multi-Layer Perceptron (MLP) takes a single 3D coordinate as input and outputs its corresponding deformation vector.

Figure 1. Overview of NIR, which is a optimization-based pair- wise medical image registration framework via neural fields. In each iteration of optimization, every position p is sampled from the coordinate of target volume T and the deformed position p′

Key technical components include:

  1. Coordinate Encoding & SIREN: Input coordinates are mapped to a higher-dimensional space using Fourier features, and the MLP utilizes sinusoidal activation functions (SIREN). This combination allows the network to overcome the "spectral bias" of standard networks and effectively capture high-frequency, fine-grained structural deformations.
  2. Two Registration Modes:
  3. Neural Displacement Field: Directly outputs displacement vectors for general deformable registration.
  4. Neural Velocity Field: Outputs a stationary velocity field, which is integrated over time using a Differentiable Ordinary Differential Equation (Neural ODE) solver. This guarantees a diffeomorphic transformation, meaning the mapping is smooth, invertible, and preserves topology (preventing anatomical structures from folding in on themselves).
  5. Hybrid Coordinate Sampler: Optimizing over every voxel in a 3D volume is memory-intensive. The authors discovered a fundamental trade-off between different stochastic sampling strategies: "downsize sampling" yields high alignment accuracy but poor local regularity, while "mini-patch sampling" preserves topology but struggles with global alignment. To solve this, they propose a two-phase hybrid scheme using two stacked neural fields ($NF_1$ and $NF_2$) optimized sequentially.
Figure 4. Overview of NIR with Hybrid Coordinate Sampling Scheme. The optimization is composed of two phases, in which two neural fields (NF1 and NF2) are optimized separately. In the first phase, NF1 is optimized with the downsize coordinate sam- pler (CS1) for 200 iterations. In the second phase, with the mini- patch coordinate sampler (CS2), fixed NF1 provides the initial deformations and only NF2 is optimized. During inference, NIR with hybrid coordinate sampler requires grid coordinates to pass through two neural fields in sequence to get the deformed coordi- nates.

Datasets

The framework was evaluated on two public 3D brain MR image datasets:

  • Mindboggle101: Used to evaluate surface alignment accuracy across 31 cortical regions.
  • OASIS: Consisting of 416 T1-weighted brain MRIs, used to evaluate volumetric alignment accuracy across 27 anatomical structures.

Key Insights and Ideas

  • Overfitting as a Feature: Rather than training a network to generalize across an entire dataset, NIR optimizes the network parameters specifically for each image pair. The network effectively "overfits" to the unique deformations of that pair, bypassing the generalization pitfalls of standard deep learning.
  • The Sampling Trade-off: The choice of how you sample coordinates during stochastic gradient descent (SGD) acts as an implicit regularizer. Uniformly downsampling coordinates across the volume helps the model capture global context quickly, whereas sampling dense local patches forces the model to respect local spatial derivatives (Jacobian determinants), preserving topology.

Key Results

NIR was compared against both learning-based baselines (VoxelMorph, SYM_Net) and traditional optimization tools (SyN, NiftyReg).

  • Accuracy & Regularity: The diffeomorphic variant, NIR-H-Diff (using the hybrid sampler), achieved state-of-the-art performance. On Mindboggle101, it achieved a Surface Dice score of 0.7904. On OASIS, it achieved a Volumetric Dice score of 0.8382 while maintaining an exceptionally low rate of topological folding ($J_{\le 0} = 4.55 \times 10^{-6}$).
  • Efficiency: NIR is highly memory-efficient, requiring less than 3500MB of GPU memory during optimization—less than half of what learning-based models require for training.
  • Speed: While slower than feed-forward deep learning inference, NIR is dramatically faster than traditional optimization methods, completing a 100-iteration optimization block in roughly 9 seconds for displacement fields and 64 seconds for velocity fields.

Limitations and Future Work

The primary limitation of NIR is its optimization runtime. Because it must optimize a new neural network from scratch for every single image pair, it cannot match the near-instantaneous inference speeds of fully supervised, feed-forward registration networks.

To address this, the authors suggest several future directions:

  • Designing adaptive coordinate samplers that focus computation on hard-to-align regions with large errors.
  • Using a learning-based model to provide a rapid initial global alignment, followed by NIR for fine-grained, patient-specific optimization.
  • Integrating shape/surface-based registration metrics alongside intensity-based metrics within the same neural field framework.

Summary updated

Abstract

Image registration is an essential step in many medical image analysis tasks. Traditional methods for image registration are primarily optimization-driven, finding the optimal deformations that maximize the similarity between two images. Recent learning-based methods, trained to directly predict transformations between two images, run much faster, but suffer from performance deficiencies due to model generalization and the inefficiency in handling individual image specific deformations. Here we present a new neural net based image registration framework, called NIR (Neural Image Registration), which is based on optimization but utilizes deep neural nets to model deformations between image pairs. NIR represents the transformation between two images with a continuous function implemented via neural fields, receiving a 3D coordinate as input and outputting the corresponding deformation vector. NIR provides two ways of generating deformation field: directly output a displacement vector field for general deformable registration, or output a velocity vector field and integrate the velocity field to derive the deformation field for diffeomorphic image registration. The optimal registration is discovered by updating the parameters of the neural field via stochastic gradient descent. We describe several design choices that facilitate model optimization, including coordinate encoding, sinusoidal activation, coordinate sampling, and intensity sampling. Experiments on two 3D MR brain scan datasets demonstrate that NIR yields state-of-the-art performance in terms of both registration accuracy and regularity, while running significantly faster than traditional optimization-based methods.
BibTeX
@article{sun2024medical,
  author = {Shanlin Sun and Kun Han and Chenyu You and Hao Tang and Deying Kong and Junayed Naushad and Xiangyi Yan and Haoyu Ma and Pooya Khosravi and James S. Duncan and Xiaohui Xie},
  title = {{Medical Image Registration via Neural Fields}},
  journal = {Medical Image Analysis},
  year = {2024},
  doi = {10.1016/j.media.2024.103249},
}
Figure