A two-step deep learning method for 3DCT-2DUS kidney registration during breathing

Yanling Chi, Yuyu Xu, Huiying Liu, Xiaoxiang Wu, Zhiqiang Liu, Jiawei Mao, Guibin Xu, Weimin Huang

Abstract

This work introduces KidneyRegNet, a novel deep registration pipeline designed for the real-time alignment of 3D CT volumes and 2D ultrasound (U/S) sequences during free breathing. The system utilizes a dual-network approach: a feature network (ULBNet) with handcrafted texture layers to bridge the semantic gap between modalities, and a hierarchical encoder-decoder registration network. By employing a "one-cycle" unsupervised transfer learning strategy, the model adapts to patient-specific breathing patterns in minutes. Experimental results on a large dataset of 132 U/S sequences and 274 CT images demonstrate a mean contour distance (MCD) of 0.94 mm for CT-to-US registration, significantly outperforming existing state-of-the-art methods.

Introduction

In image-guided interventions, aligning a high-resolution pre-operative 3D CT scan with live 2D ultrasound is critical for accurate targeting. However, 3DCT-2DUS registration faces significant challenges: the dimensionality gap between 3D and 2D data, the modality gap (differing appearances of CT and U/S), and the continuous deformation caused by respiratory motion. Traditional methods often fail to provide the speed or accuracy required for clinical environments. KidneyRegNet addresses these hurdles through a specialized architecture and a two-step training paradigm.

Methods

1. KidneyRegNet Architecture

The pipeline consists of two primary components designed to handle the complexities of multi-modal registration:

  • Feature Network (ULBNet): This 5-level U-Net extracts shared feature maps from both CT and U/S inputs. It incorporates Local Binary Convolution (LBC) layers to focus on anatomical textures, effectively normalizing the disparate visual styles of the two modalities into a common mathematical representation.
  • Registration Network: Utilizing an encoder-decoder structure, this network performs hierarchical regression. Rather than predicting a single global transformation, it estimates parameters across multiple resolutions. This prevents the model from settling into local optima and ensures stability during the registration of moving organs.
The KidneyRegNet architecture, illustrating the hierarchical encoder-decoder structure and feature extraction components.

2. Training and Loss Functions

The model is trained using a Feature-Image-Motion (FIM) loss function. This composite loss includes Dice coefficients for shape alignment, Modality Independent Neighbourhood Descriptors (MIND) for local detail, and smoothing constraints to ensure the predicted motion remains physiologically plausible.

To overcome the scarcity of paired CT-US data, the authors employed a two-step training strategy:

  1. Pre-training: The model is trained on synthetic data generated by modeling kidney motion with Gaussian distributions.
  2. One-Cycle Transfer Learning: Upon clinical application, the model is refined using the first breathing cycle of the specific patient. This "tune-up" reduces on-site preparation time from 46 minutes to approximately 2–3 minutes.

Key Results

KidneyRegNet was evaluated against several state-of-the-art (SOTA) models, including VoxelMorph and C2FViT. The proposed method achieved a Mean Contour Distance (MCD) of 0.94 mm for CT-to-US registration and 1.15 mm for CT-to-reference-CT registration.

Comparative performance metrics (HD and MCD) for KidneyRegNet versus state-of-the-art registration models across one-step and two-step learning.

As shown in the results table, the two-step learning approach (incorporating patient-specific transfer learning) significantly reduces error compared to one-step learning. KidneyRegNet consistently outperformed other models, particularly in scenarios involving large respiratory transformations, where it maintained sub-millimeter precision.

Conclusion

KidneyRegNet provides a robust solution for real-time 3DCT-2DUS registration. By combining hierarchical regression with a rapid unsupervised transfer learning strategy, the method achieves the sub-millimeter accuracy necessary for high-precision medical applications. Future work will focus on improving robustness against poor initial segmentations and incorporating non-rigid deformation modeling to further enhance registration fidelity during complex breathing cycles.

Summary updated

Abstract

This work proposed KidneyRegNet, a novel deep registration pipeline for 3D CT and 2D U/S kidney scans of free breathing, which comprises a feature network, and a 3D-2D CNN-based registration network. The feature network has handcrafted texture feature layers to reduce the semantic gap. The registration network is an encoder-decoder structure with loss of feature-image-motion (FIM), which enables hierarchical regression at decoder layers and avoids multiple network concatenation. It was first pretrained with a retrospective dataset cum training data generation strategy and then adapted to specific patient data under unsupervised one-cycle transfer learning in onsite applications. The experiment was performed on 132 U/S sequences, 39 multiple-phase CT and 210 public single-phase CT images, and 25 pairs of CT and U/S sequences. This resulted in a mean contour distance (MCD) of 0.94 mm between kidneys on CT and U/S images and MCD of 1.15 mm on CT and reference CT images. Datasets with small transformations resulted in MCDs of 0.82 and 1.02 mm, respectively. Large transformations resulted in MCDs of 1.10 and 1.28 mm, respectively. This work addressed difficulties in 3DCT-2DUS kidney registration during free breathing via novel network structures and training strategies.
BibTeX
@inproceedings{chi2023two,
  author = {Yanling Chi and Yuyu Xu and Huiying Liu and Xiaoxiang Wu and Zhiqiang Liu and Jiawei Mao and Guibin Xu and Weimin Huang},
  title = {{A two-step deep learning method for 3DCT-2DUS kidney registration during breathing}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-40133-5},
}
Figure