H-ViT: A Hierarchical Vision Transformer for Deformable Image Registration

Morteza Ghahremani, Mohammad Khateri, Bailiang Jian, Benedikt Wiestler, Ehsan Adeli, Christian WachingerCVPR 2024 · DOI

Introduction

Deformable image registration is a fundamental task in medical imaging, involving the non-linear mapping of a "moving" image to a "target" atlas. This process is essential for longitudinal studies and cross-patient anatomical comparisons. While deep learning has advanced the field, traditional Convolutional Neural Networks (CNNs) often struggle with long-range spatial dependencies due to their limited receptive fields. Conversely, standard Vision Transformers (ViTs) provide global context but often fail to capture the hierarchical nature of deformation fields.

H-ViT addresses these challenges by introducing a hierarchical dual-attention mechanism. This approach combines self-attention and cross-attention to capture multi-scale deformation patterns, allowing fine-grained image patches to query abstract flow features from higher scales.

Methodology

The H-ViT architecture is a hybrid CNN-Transformer model designed with a "top-down" representation strategy.

The H-ViT architecture featuring a CNN backbone and a Dual-Attention Block for hierarchical feature extraction.

The model consists of three primary components:

  1. CNN Backbone: Utilizing a Feature Pyramid Network (FPN)-like structure, it extracts multi-scale features from the concatenated moving and target images.
  2. Dual-Attention Block: This core innovation applies self-attention within each scale to capture short-range dependencies and hierarchical cross-attention between scales. In the cross-attention phase, query tokens from finer scales interact with key and value tokens from coarser scales, enabling "big picture" flow patterns to guide local deformation estimation.
  3. Diffeomorphic Mapping: To ensure physically plausible, smooth, and invertible deformations, the model employs a stationary velocity field approach with scaling and squaring.

Datasets and Evaluation

The model underwent rigorous evaluation across five publicly available T1-weighted brain MRI databases: OASIS (primary inter-patient benchmarking), IXI (training and multi-mode testing), and ADNI, LPBA, and Mindboggle (generalization testing).

Key Insights

  • Hierarchical Flow Representation: By using cross-attention to look "up" the feature pyramid, H-ViT recognizes that local voxel movements are often components of larger anatomical shifts.
  • Computational Efficiency: Selective cross-attention from coarser feature maps avoids the prohibitive cost of global self-attention at high resolutions.
  • Translation Invariance: The hybrid design maintains the translation invariance strengths of CNNs, which is critical for structural consistency in medical imaging.

Results

H-ViT consistently outperformed both CNN-based baselines (e.g., VoxelMorph) and Transformer-based models (e.g., TransMorph).

Quantitative comparison of H-ViT against state-of-the-art registration methods showing superior Dice scores.

Key performance highlights include:

  • Accuracy: H-ViT achieved a Dice score of 0.810 on the IXI dataset and 0.876 on OASIS, marking a significant improvement over previous state-of-the-art (SOTA) methods.
  • Visual Fidelity: Qualitative assessments revealed superior warping in complex anatomical regions like the frontal gyrus and temporal lobes.
  • Robustness: The model demonstrated excellent generalization when trained on one dataset and tested on others with different population characteristics.

Limitations and Future Directions

Despite its performance, H-ViT introduces higher computational overhead, requiring approximately 22.6 GB of VRAM. Future research could focus on extending this mechanism to other modalities like CT or more highly deformable organs such as the lungs or liver. Additionally, the model's sensitivity to hyperparameters like patch size suggests a need for automated tuning strategies in diverse clinical applications.

Summary updated

BibTeX
@inproceedings{ghahremani2024h,
  author = {Morteza Ghahremani and Mohammad Khateri and Bailiang Jian and Benedikt Wiestler and Ehsan Adeli and Christian Wachinger},
  title = {{H-ViT: A Hierarchical Vision Transformer for Deformable Image Registration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01094},
}
Figure