Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions

Xin Fan, Zi Li, Xiaolin Wang, Risheng Liu, Zhongxuan Luo, Hao Huang

TL;DR

AutoReg is a new framework that automates the design of deep learning models for medical image registration. By jointly optimizing the network architecture and the loss function hyperparameters through a triple-level optimization strategy, it removes the need for manual expert tuning and achieves state-of-the-art accuracy with significantly faster inference times.

What Problem Does This Paper Address?

Deformable image registration (DIR)—the process of aligning two medical images by finding a non-linear spatial transformation—is a cornerstone of medical AI. While deep learning has made registration much faster than traditional iterative methods, it introduced a new bottleneck: "expert-in-the-loop" tuning.

Every new clinical scenario (e.g., aligning brain MRIs vs. lung CTs, or T1-weighted vs. T2-weighted images) requires a different network architecture and a carefully balanced loss function. Currently, computer vision experts spend weeks manually testing different convolutional layers and weighting factors to get optimal results. This "manual train-trial" process is a major barrier for clinical researchers who have data but lack deep learning engineering expertise.

Methods

The authors propose AutoReg, which treats the design of a registration algorithm as a triple-level optimization problem:

  1. Level 1 (Weights $\omega$): Standard training of the network parameters on a training set.
  2. Level 2 (Architecture $\alpha$): Searching for the best convolutional operators (e.g., dilated vs. separable convolutions) using a validation set.
  3. Level 3 (Hyperparameters $\lambda$): Finding the optimal weights for the loss function terms (similarity vs. regularization) based on anatomical overlap (Dice scores) in the validation set.
The AutoReg framework automates the design process that previously required human experts.

To make this computationally feasible, they use a differentiable Neural Architecture Search (NAS). Instead of picking one operation, the model learns a weighted combination of eight candidate operations (like $3\times3\times3$ convolutions or dilated convolutions) and gradually "relaxes" the choice until one dominant path emerges. They also introduce a unified loss function that combines intensity-based metrics (NCC) with structural descriptors (MIND), making the system robust to multi-modal data.

The search space consists of Adaptive Feature Cells and Task-aware Deformation Cells.

Datasets

The authors validated AutoReg across a diverse range of scenarios to prove its "self-adapting" capability:

  • Brain MRI: ADNI, ABIDE, PPMI, and OASIS (T1-to-Atlas and T1-to-T1).
  • Multi-modal Brain: BraTS18 and ISeg19 (T1-to-T2 and T2-to-T1).
  • Knee MRI: Osteoarthritis Initiative (OAI) dataset.
  • Lung CT: Learn2Reg challenge (Inspiration-to-Expiration).

Key Insights and Ideas

  • Heterogeneous Architecture is Key: The search revealed that different tasks prefer different "cells." For example, knee registration (which involves large deformations) favors different convolutional kernels than brain registration.
  • Receptive Field Matters: The model tended to select convolutions with large receptive fields (like dilated convolutions) for high-resolution layers, confirming that capturing long-range spatial context is vital for registration.
  • Automated Loss Tuning Works: By using Dice scores (anatomical overlap) as the signal for hyperparameter optimization, the model successfully learned how much to penalize "folding" in the deformation field without human intervention.
  • Efficiency Gains: The searched architectures were not just more accurate, but also more efficient. The final model is roughly half the size of a standard UNet and twice as fast during inference.
Heatmaps showing how the preferred network architecture changes based on the specific registration task.

Key Results

  • Accuracy: AutoReg achieved higher Dice scores across all datasets compared to popular baselines like VoxelMorph (VM) and SyN. For instance, on the ADNI dataset, it reached a Dice score of 0.774 compared to VM's 0.761.
  • Speed: Inference time for a 3D volume pair was reduced from 0.558 seconds (standard UNet) to 0.270 seconds.
  • Stability: As shown in the box plots below, AutoReg produced fewer outliers and lower variance in registration accuracy across test pairs compared to other methods.
  • Search Efficiency: While manual tuning can take weeks, the automated search phase takes approximately 3 days on a single GPU.
Box plots demonstrating that AutoReg provides more stable and accurate registration across various tasks.

Limitations and Future Work

The primary limitation is the reliance on labeled validation data. To optimize the loss function hyperparameters ($\lambda$), the system currently requires segmentation maps or landmarks to calculate Dice scores. In many clinical settings, such labels are scarce.

Future work aims to expand the search space to include more complex elements like skip connections, network depth, and multi-scale topology. The authors also suggest moving toward a more generalized framework that could handle even more highly-parameterized registration models.

Summary updated

Abstract

Deformable image registration plays a critical role in various tasks of medical image analysis. A successful registration algorithm, either derived from conventional energy optimization or deep networks requires tremendous efforts from computer experts to well design registration energy or to carefully tune network architectures for the specific type of medical data. To tackle the aforementioned problems, this paper proposes an automated learning registration algorithm (AutoReg) that cooperatively optimizes both architectures and their corresponding training objectives, enabling non-computer experts, e.g., medical/clinical users, to conveniently find off-the-shelf registration algorithms for diverse scenarios. Specifically, we establish a triple-level framework to deduce registration network architectures and objectives with an auto-searching mechanism and cooperating optimization. We conduct image registration experiments on multi-site volume datasets and various registration tasks. Extensive results demonstrate that our AutoReg may automatically learn an optimal deep registration network for given volumes and achieve state-of-the-art performance, also significantly improving computation efficiency than the mainstream UNet architectures (from 0.558 to 0.270 seconds for a 3D image pair on the same configuration).
BibTeX
@misc{fan2022automated,
  author = {Xin Fan and Zi Li and Ziyang Li and Xiaolin Wang and Risheng Liu and Zhongxuan Luo and Hao Huang},
  title = {{Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions}},
  year = {2022},
  eprint = {2203.06810},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.06810},
}
Figure