Learning deep similarity metric for 3D MR-TRUS image registration

Grant Haskins, Jochen Kruecker, Uwe Kruger, Sheng Xu, Peter A. Pinto, Bradford J. Wood, Pingkun Yan

TL;DR

This paper introduces a novel deep learning-based similarity metric for registering 3D MR and TRUS images, crucial for accurate prostate cancer biopsies. By learning a direct measure of registration error, their method significantly outperforms traditional metrics like mutual information and MIND descriptors, achieving more robust and accurate alignments, especially under challenging initial conditions.

What Problem Does This Paper Address?

Prostate cancer diagnosis and treatment often rely on guiding biopsies with imaging. While multi-parametric MRI offers high sensitivity for detecting cancerous lesions, transrectal ultrasound (TRUS) is the standard for guiding biopsies due to its real-time feedback and cost-effectiveness. Fusing these two modalities (MR-TRUS fusion) promises to improve biopsy accuracy by combining the strengths of both. However, a critical bottleneck in this fusion process is image registration – aligning the MR and TRUS images into a common coordinate space. This is particularly challenging due to the significant appearance differences between MR and TRUS images, stemming from their distinct imaging physics. Existing registration methods often struggle with robustness, especially when initial alignments are poor, leading to inaccurate targeting and potentially missed cancers. This paper aims to address this by developing a more robust and accurate similarity metric for MR-TRUS registration.

Methods

The core of this work is a deep convolutional neural network (CNN) designed to learn a similarity metric directly. Instead of relying on traditional hand-crafted features or statistical relationships, the CNN takes a pair of 3D MR and TRUS images as input and directly outputs an estimate of the target registration error (TRE). This TRE serves as the similarity metric, quantifying how well the images are aligned.

Key methodological contributions include:

  • Deep Similarity Metric Learning: A custom 9-layer 3D CNN architecture is proposed. It utilizes skip connections to fuse features from different network levels, enhancing its ability to capture relevant information. The network is trained to regress the TRE, effectively learning what constitutes a good alignment.
  • Composite Optimization Strategy (DINO): Recognizing that learned similarity metrics can be non-convex and non-smooth, the authors propose a two-stage optimization strategy called DINO (Differential Evolution initialized Newton-based optimization).
  • Differential Evolution: This global search algorithm is used first to explore the solution space and find a broad region of good alignment, thus expanding the "capture range" of the registration.
  • Newton-based Optimization (BFGS): Once a promising region is found by differential evolution, a local, second-order optimization method (BFGS) is employed to refine the alignment to a high degree of accuracy.
  • Multi-pass Approach: To further improve robustness and smooth the objective function for optimization, a multi-pass approach is used. This involves slightly perturbing the moving image multiple times and averaging the TRE estimates to get a more stable evaluation of the similarity metric.

Datasets

The study utilized a dataset of 679 MR-TRUS image pairs acquired from MR-TRUS fusion-guided prostate cancer biopsy procedures at the National Institute of Health (NIH). Each set included a T2-weighted MR volume and a reconstructed 3D ultrasound volume. The data was split into training (539 cases), validation (70 cases), and test sets (70 cases). The authors generated training data by applying known rigid perturbations to the ground truth registration, allowing them to compute the corresponding TRE as the ground truth label for training the CNN.

Key Insights and Ideas

  • Learning Registration Error Directly is Powerful: The most significant insight is that a CNN can be trained to directly estimate the target registration error (TRE), serving as a highly effective similarity metric for multimodal registration. This bypasses the limitations of traditional metrics that rely on assumptions about image statistics or feature correspondences.
  • Deep Metrics Need Robust Optimization: The learned similarity metric is inherently non-convex and non-smooth, making it susceptible to getting stuck in local minima. This underscores the necessity of advanced optimization strategies like DINO, which combine global exploration with local refinement.
  • Capture Range is Crucial for Clinical Robustness: The DINO strategy, particularly the differential evolution component, significantly expands the capture range of the registration. This means the method is less sensitive to the initial alignment, a critical factor for practical clinical applications where manual initialization might not be perfect.
  • Multi-pass Smoothing Enhances Metric Reliability: Averaging TRE estimates over small perturbations provides a smoother, more reliable objective function for the optimizer, leading to more consistent results.

Key Results

  • Superior Accuracy: The proposed deep similarity metric with the DINO optimizer achieved a mean TRE of 3.86mm, significantly outperforming mutual information (mean TRE of 8.96mm with DINO) and MIND descriptors (mean TRE of 6.42mm with DINO) on the test set, even with an initial TRE of 16mm.
  • Robustness to Initialization: The DINO approach demonstrated robustness to poor initializations, achieving accurate registrations even when starting with an 8mm or 16mm TRE.
  • Multi-pass Improvement: The multi-pass approach further improved the deep metric's performance, reducing the mean TRE to 3.82mm compared to the single-pass approach (3.97mm) when using DINO.
  • Qualitative Improvements: Visualizations (Fig. 4) show that the registration results using the learned metric are visually more aligned and accurate compared to MI and MIND-based methods.

Limitations and Future Work

  • Rigid Transformation Model: The current work uses a rigid transformation model. The authors acknowledge that this limits registration accuracy due to potential deformation of the prostate between image acquisitions.
  • Future Work: The authors plan to investigate deformable registration models to better handle prostate deformation. They also aim to explore variants of their CNN training strategy, further speed up the registration process, and encourage more computationally efficient exploration of the solution space. They also suggest that their software, which allows manual override and provides instant feedback on registration quality, could facilitate faster clinical adoption.

Summary updated

Abstract

Purpose: The fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images for guiding targeted prostate biopsy has significantly improved the biopsy yield of aggressive cancers. A key component of MR-TRUS fusion is image registration. However, it is very challenging to obtain a robust automatic MR-TRUS registration due to the large appearance difference between the two imaging modalities. The work presented in this paper aims to tackle this problem by addressing two challenges: (i) the definition of a suitable similarity metric and (ii) the determination of a suitable optimization strategy. Methods: This work proposes the use of a deep convolutional neural network to learn a similarity metric for MR-TRUS registration. We also use a composite optimization strategy that explores the solution space in order to search for a suitable initialization for the second-order optimization of the learned metric. Further, a multi-pass approach is used in order to smooth the metric for optimization. Results: The learned similarity metric outperforms the classical mutual information and also the state-of-the-art MIND feature based methods. The results indicate that the overall registration framework has a large capture range. The proposed deep similarity metric based approach obtained a mean TRE of 3.86mm (with an initial TRE of 16mm) for this challenging problem. Conclusion: A similarity metric that is learned using a deep neural network can be used to assess the quality of any given image registration and can be used in conjunction with the aforementioned optimization framework to perform automatic registration that is robust to poor initialization.
BibTeX
@article{haskins2019learning,
  author = {Grant Haskins and Jochen Kruecker and Uwe Kruger and Sheng Xu and Peter A. Pinto and Bradford J. Wood and Pingkun Yan},
  title = {{Learning deep similarity metric for 3D MR-TRUS image registration}},
  journal = {Int. J. Comput. Assist. Radiol. Surg},
  year = {2019},
  doi = {10.1007/s11548-018-1875-7},
}
Figure