Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing

Hang Zhang, Xiang Chen, Renjiu Hu, Rongguang Wang, Jinwei Zhang, Min Liu, Yaonan Wang, Gaolei Li, Xinxing Cheng, Jinming Duan

Introduction

Deformable Image Registration (DIR) is a critical task in medical imaging, involving the determination of a non-linear spatial mapping between two images. While deep learning has significantly accelerated DIR, unsupervised methods—which operate without ground-truth labels—frequently struggle in two specific scenarios: the aperture problem and large displacements.

In homogeneous or textureless regions, such as the spaces between retinal vessels, local evidence is often insufficient to determine movement. Furthermore, when structures move further than their own width, standard convolutional filters often lose the signal. This paper argues that current methods fail because they treat smoothness regularization merely as a loss term during training, leaving the forward pass unconstrained. To address this, the authors introduce SmoothProper, a plug-and-play neural module that integrates structural smoothing directly into the network's forward pass.

The SmoothProper Module

SmoothProper is an unrolled neural network layer grounded in variational optimization. It treats registration as a bi-level optimization problem where the "inner loop" (SmoothProper) ensures the predicted deformation field is smooth and structurally consistent during every forward pass.

Key Architectural Features:

  • Deep Unrolling: The module iteratively updates the flow field using a coordinate descent approach (typically $K=6$ iterations), mimicking classical iterative optimization within a differentiable architecture.
  • Basis-Constrained Formulation: By learning "dictionary atoms" or basis vectors, the deformation is represented as a linear combination of learned patterns. This creates a "smooth-reinforce" interplay: smoothing steps diffuse signals across gaps, while reinforcement steps "lock in" strong signals to maintain field integrity.
  • Conditional Hypernetwork: To eliminate manual tuning of the regularization hyperparameter ($\lambda$), a small MLP generates basis vectors conditioned on $\lambda$, allowing the model to adapt to varying smoothness requirements automatically.

Experimental Evaluation

The researchers evaluated SmoothProper on the Fundus Image Registration (FIRE) dataset, which consists of 134 image pairs of retinal vessels at a high resolution of 2912 × 2912 pixels.

Retinal vessel registration challenges: (a) Full fundus image with regions of interest (b-d) showing sparse vessel structures and homogeneous backgrounds.

Retinal images represent a significant challenge for DIR because vessels are thin and sparse, often undergoing large displacements or anatomical changes between scans. SmoothProper significantly outperformed both traditional descriptor-matching and state-of-the-art deep learning methods:

  • Accuracy: It achieved a Target Registration Error (TRE) of 1.88 pixels, compared to 4.97 pixels for the next best deep unrolling method (VR-Net).
  • Robustness: The method reduced registration error by over 67% compared to RetinaIPA, a specialized detector-based method.
  • Efficiency: Despite the iterative inner loop, the entire registration process takes under one second per scan pair on an NVIDIA A100.

Discussion and Future Work

SmoothProper reframes regularization as a "message passing" mechanism that propagates flow information from feature-rich areas (vessel edges) into feature-poor areas. By embedding optimization logic into the architecture itself, the model maintains structural integrity even on unseen data.

While highly effective, the method currently relies on an initial global alignment (affine or homography) provided by descriptor-matching. Future work includes extending this 2D logic to 3D volumes, such as MRI or CT scans, and further refining the automated selection of the $\lambda$ range during training.

Summary updated

BibTeX
@inproceedings{zhang2025unsupervised,
  author = {Hang Zhang and Xiang Chen and Renjiu Hu and Rongguang Wang and Jinwei Zhang and Min Liu and Yaonan Wang and Gaolei Li and Xinxing Cheng and Jinming Duan},
  title = {{Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing}},
  booktitle = {IPMI},
  year = {2025},
  doi = {10.1007/978-3-031-96628-6\_8},
}
Figure