Decoder-Only Image Registration

Xi Jia, Wenqi Lu, Xinxing Cheng, Jinming Duan

TL;DR

This paper introduces LessNet, a novel decoder-only deep learning architecture for unsupervised medical image registration. By eliminating the learnable encoder and replacing it with handcrafted features, LessNet achieves comparable registration accuracy to state-of-the-art methods while significantly reducing computational resources, training time, and memory usage.

What Problem Does This Paper Address?

Medical image registration is crucial for tasks like disease diagnosis and treatment planning, aiming to align images from different sources or times. Traditional methods rely on iterative optimization, which can be slow and require extensive hyperparameter tuning. Deep learning approaches have emerged as faster alternatives, typically using encoder-decoder architectures to predict dense displacement fields. However, these encoder-decoder networks often have large parameter counts and high computational demands, especially for 3D medical images. This paper questions the necessity of a learnable encoder in these architectures, hypothesizing that a well-designed decoder, potentially augmented with handcrafted features, could suffice, leading to more efficient registration models.

Methods

The core innovation of LessNet is its decoder-only architecture. Instead of a traditional encoder that learns hierarchical features from input images, LessNet utilizes simple, handcrafted features derived from multi-scale pooling operations (max, average, and min pooling) applied to the input image pair. These pooling operations generate feature maps at different spatial resolutions.

The architecture then consists of a learnable convolutional decoder that takes these handcrafted features, along with the original input images (concatenated at appropriate stages), to predict a dense displacement field. The decoder is structured into four hierarchical convolutional blocks, progressively upsampling and refining the feature maps. The authors explored two loss functions: a standard similarity and regularization loss for direct displacement prediction, and a modified loss for predicting stationary velocity fields to achieve diffeomorphic registration.

Datasets

The paper evaluates LessNet on two publicly available brain MRI datasets:

  • OASIS-1 dataset: Used for subject-to-subject (inter-subject) registration. A processed version was used, containing 2D slices from 3D volumes. The dataset was split into training, validation, and testing sets, resulting in a large number of image pairs.
  • IXI dataset: Used for atlas-to-subject registration. A processed version was used, and the evaluation protocol followed that of previous work. This dataset involved 3D volumes.

Key Insights and Ideas

  • Encoder Redundancy: The paper provides experimental evidence (Table I) suggesting that the learnable encoder in traditional registration networks might be redundant. Freezing the encoder and training only the decoder still yields significant registration accuracy, implying that the decoder plays a more critical role.
  • Handcrafted Features Suffice: The success of LessNet demonstrates that simple, computationally inexpensive handcrafted features (like multi-scale pooling) can effectively replace a complex, learnable encoder for image registration. This challenges the trend towards ever-larger and more complex network architectures.
  • Efficiency Gains: By eliminating the learnable encoder, LessNet drastically reduces parameter count, memory usage, and computational operations (mult-adds), leading to faster training and inference.
  • Flexibility: The decoder-only framework is presented as adaptable to different loss functions, allowing for both direct displacement prediction and diffeomorphic registration.

Key Results

  • Comparable Accuracy: LessNet achieved registration accuracy (Dice score) comparable to state-of-the-art methods like VoxelMorph and TransMorph on both OASIS-1 and IXI datasets. For instance, on OASIS-1, LessNet16 achieved a Dice score of 0.768±0.039, similar to TransMorph (0.768±0.039).
  • Significant Efficiency Improvements:
    • Parameters: LessNet16 used only 2.11% of the parameters compared to TransMorph.
  • Mult-Adds: LessNet16 required 73.33% of the mult-adds of TransMorph.
  • Memory: LessNet achieved significantly lower GPU memory usage compared to methods like TransMorph.
  • Training Time: LessNet achieved its performance with considerably less GPU runtime (Fig. 1).
  • Diffeomorphic Registration: The diffeomorphic version of LessNet (Diff-LessNet) also performed well, achieving nearly zero negative Jacobian determinants, indicating realistic deformations. On the IXI dataset, Diff-LessNet6 outperformed Diff-VoxelMorph and Diff-TransMorph.

Limitations and Future Work

  • Handcrafted Features: While pooling features were used as a proof of concept, the authors acknowledge that other handcrafted features or features from pre-trained networks could be explored to potentially further improve performance.
  • Dataset Specificity: The paper notes that while LessNet performed well on the tested datasets, the observation that diffeomorphic versions might lag behind non-diffeomorphic ones in Dice score on OASIS-1 might not be universally true for all datasets.
  • Future Directions: The authors encourage experimentation with alternative handcrafted or pre-trained features to enhance LessNet's capabilities. They also suggest that the decoder-only paradigm could be extended to other image registration tasks.

Summary updated

Abstract

In unsupervised medical image registration, the predominant approaches involve the utilization of a encoder-decoder network architecture, allowing for precise prediction of dense, full-resolution displacement fields from given paired images. Despite its widespread use in the literature, we argue for the necessity of making both the encoder and decoder learnable in such an architecture. For this, we propose a novel network architecture, termed LessNet in this paper, which contains only a learnable decoder, while entirely omitting the utilization of a learnable encoder. LessNet substitutes the learnable encoder with simple, handcrafted features, eliminating the need to learn (optimize) network parameters in the encoder altogether. Consequently, this leads to a compact, efficient, and decoder-only architecture for 3D medical image registration. Evaluated on two publicly available brain MRI datasets, we demonstrate that our decoder-only LessNet can effectively and efficiently learn both dense displacement and diffeomorphic deformation fields in 3D. Furthermore, our decoder-only LessNet can achieve comparable registration performance to state-of-the-art methods such as VoxelMorph and TransMorph, while requiring significantly fewer computational resources. Our code and pre-trained models are available at https://github.com/xi-jia/LessNet.
BibTeX
@article{duan2025decoder,
  author = {Jinming DUAN and Xi Jia and Wenqi Lu and Xinxing Cheng and Jinming Duan},
  title = {{Decoder-Only Image Registration}},
  journal = {IEEE TMI},
  year = {2025},
  doi = {10.1109/tmi.2025.3562056},
}
Figure