TransMorph: Transformer for unsupervised medical image registration

Junyu Chen, Yong Du, Yufan He, William Paul Segars, Ye Li, Eric C. FreyMedical Image Analysis 2021 · DOI · OpenReview

TL;DR

This paper introduces TransMorph, a novel hybrid Transformer-Convolutional Neural Network (CNN) model for unsupervised medical image registration. TransMorph leverages the power of Transformers to capture long-range spatial relationships, leading to improved registration accuracy and robustness, particularly for challenging cases with large deformations.

What Problem Does This Paper Address?

Deformable image registration (DIR) is a crucial task in medical imaging, enabling the alignment of images to understand anatomical correspondence. Traditional DIR methods are computationally expensive and slow, requiring optimization for each new image pair. While deep learning, particularly CNNs, has accelerated this process, CNNs struggle to capture long-range spatial dependencies due to their inherent locality. Vision Transformers (ViTs) have shown promise in computer vision by effectively modeling these long-range relationships. This paper aims to bridge this gap by applying Transformer architectures to medical image registration, specifically addressing the challenge of accurately aligning images with significant deformations.

Methods

TransMorph is a hybrid model that combines the strengths of Swin Transformers and CNNs. The core idea is to use a Swin Transformer as an encoder to capture global spatial context and long-range dependencies from the input images. This is followed by a CNN decoder that processes the Transformer's output to generate a dense displacement field.

Key methodological contributions include:

  • Hybrid Transformer-CNN Architecture: A Swin Transformer encoder is used to extract features, followed by a CNN decoder to produce the deformation field.
  • Unsupervised Learning: The model is trained in an unsupervised manner, relying on image similarity and deformation regularization losses, eliminating the need for ground-truth deformation fields.
  • Diffeomorphic Variants: The authors propose integrating TransMorph into existing frameworks to ensure topology-preserving (diffeomorphic) deformations, crucial for maintaining anatomical integrity.
  • Bayesian Uncertainty Quantification: A Bayesian variant of TransMorph is introduced to provide well-calibrated estimates of registration uncertainty, indicating the reliability of the predicted deformation.
  • Architecture Analysis: The paper investigates the necessity of positional embeddings in Transformers for image registration, concluding they are not required and can even be detrimental.

Datasets

The proposed TransMorph model was evaluated on three distinct datasets:

  • Inter-patient Brain MRI Registration: A dataset of 260 T1-weighted brain MRI images acquired at Johns Hopkins University.
  • Atlas-to-patient Brain MRI Registration: Utilized the Information eXtraction from Images (IXI) database, with 576 T1-weighted brain MRI images.
  • XCAT-to-CT Registration: A novel application using a four-dimensional extended cardiac-torso (XCAT) phantom and a CT dataset from Duke University, aiming to create anatomically variable phantoms.

Key Insights and Ideas

  • Transformers Excel at Long-Range Dependencies: The study strongly suggests that Transformers, with their inherent ability to capture long-range spatial relationships, are a superior choice for image registration compared to traditional CNNs, especially when dealing with large deformations.
  • Positional Embeddings are Unnecessary for Registration: Contrary to common practice in other vision tasks, positional embeddings in Transformers were found to be redundant for image registration. The implicit learning of positional information through the decoder and loss backpropagation is sufficient.
  • Hybrid Models Offer the Best of Both Worlds: Combining the global context-aware capabilities of Transformers with the local feature extraction of CNNs in a hybrid architecture proves highly effective for medical image registration.
  • Diffeomorphic and Bayesian Variants Enhance Robustness: The ability to easily integrate diffeomorphic constraints and obtain uncertainty estimates significantly enhances the practical utility and trustworthiness of the registration model.

Key Results

  • State-of-the-Art Performance: TransMorph achieved the highest mean Dice scores across various registration tasks, outperforming both traditional and existing deep learning methods. For inter-patient brain MRI registration, it achieved a mean Dice score of 0.745.
  • Reduced Foldings: The proposed models, particularly the diffeomorphic variants, generated deformations with almost no foldings (0% of |J| ≤ 0), indicating high topological consistency.
  • Transformer Superiority: Transformer-based models generally outperformed CNN-based models, with TransMorph showing superior performance even compared to other Transformer architectures.
  • Computational Efficiency: Despite its advanced architecture, TransMorph demonstrated a moderate computational complexity, significantly less than some other Transformer-based models, and achieved faster convergence during training.
  • Effective Uncertainty Estimation: The Bayesian variant of TransMorph provided well-calibrated appearance uncertainty estimates, accurately reflecting registration errors.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that extensive grid searches for optimal hyperparameters for baseline methods were not performed due to computational constraints.
  • Multi-modal Registration: While the current work focuses on mono-modal registration, the authors suggest exploring alternative loss functions, such as mutual information, to expand the method's potential for multi-modal registration tasks.
  • Appearance Uncertainty Metric: For multi-modal registration, MSE might not be the optimal metric for appearance uncertainty. The authors suggest that predictive variance might be a more suitable choice in such scenarios.

Summary updated

Abstract

In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.
BibTeX
@article{chen2021transmorph,
  author = {Junyu Chen and Yong Du and Yufan He and William Paul Segars and Ye Li and Eric C. Frey},
  title = {{TransMorph: Transformer for unsupervised medical image registration}},
  journal = {Medical Image Analysis},
  year = {2021},
  doi = {10.1016/j.media.2022.102615},
}
Figure