Label-driven weakly-supervised learning for multimodal deformarle image registration

Yipeng Hu, Marc Modat, Eli Gibson, Nooshin Ghavami, Ester Bonmati, Caroline M. Moore, Mark Emberton, J. Alison Noble, Dean C. Barratt, Tom Vercauteren

TL;DR

Researchers have developed a weakly-supervised deep learning framework that registers different types of medical images (like MR and Ultrasound) by learning from anatomical labels during training. By bypassing the need for traditional "similarity measures," the system achieves real-time, fully automatic deformable registration with high accuracy, even when the images look nothing alike.

What Problem Does This Paper Address?

In prostate cancer treatment, doctors often use two types of images: MRI, which shows tumors clearly but is taken before surgery, and Transrectal Ultrasound (TRUS), which is used during surgery but has poor contrast. To guide the surgeon, these images must be "registered" (aligned) so the MRI data can be overlaid on the live ultrasound.

This is incredibly difficult for two reasons:

  1. No Shared Language: MRI and Ultrasound intensities don't map to each other linearly. Traditional algorithms struggle to find a "similarity measure" to tell if two pixels represent the same tissue.
  2. Deformation: The prostate is soft; it moves and changes shape between the MRI scan and the surgery.

Existing methods either require slow manual landmarking during surgery or rely on simulated data that doesn't reflect real-world anatomy.

Methods

The authors propose a label-driven, weakly-supervised approach. Instead of telling the network how pixels should move (which is hard to know), they tell the network to move the "labels" (segmentations of organs, vessels, or cysts) from the MRI to match the labels on the Ultrasound.

Key Technical Components: * Weak Supervision: Anatomical labels (like the prostate gland, urethra, or lesions) are used as proxies for voxel-level correspondence. These labels are only needed during training. At "inference" (in the clinic), the network only needs the raw images. * Dual-Network Architecture: To handle large movements, the system uses two connected networks: * Global-Net: Predicts a 12-degree-of-freedom affine transformation (rotation, scale, translation). * Local-Net: A 3D ResNet-style architecture that predicts a dense displacement field (DDF) for fine-grained, non-rigid warping. * One-Sided Label Smoothing: To prevent the network from overfitting on sharp binary masks and to increase the "capture range" (the distance from which the algorithm can successfully pull structures together), they apply a smoothing function based on an inverse distance transform.

Datasets

The study used a substantial clinical dataset:

  • 111 pairs of T2-weighted MR and 3D TRUS images from 76 prostate cancer patients.
  • 2,351 pairs of corresponding anatomical landmarks (apex, base, urethra, lesions, etc.) manually labeled by experts.
  • 4,000+ total anatomical labels were used across the dataset for training and validation.

Key Insights and Ideas

  • Labels as a Universal Language: By training the network to align higher-level structures (labels) rather than raw pixel intensities, the authors sidestep the "multimodal similarity" problem entirely. The network learns the underlying spatial relationships between the modalities implicitly.
  • The "Global-then-Local" Strategy: The authors found that predicting local deformations alone often led to "implausible" results if the initial alignment was off. By forcing the network to solve the global alignment first via the Global-Net, the Local-Net could focus on realistic tissue warping.
  • Sampling for Quality: Not all labels are created equal. The researchers sampled "high-confidence" labels (those verified by multiple experts) more frequently during training to weight the model toward the most reliable data.

Key Results

  • Accuracy: The system achieved a median Target Registration Error (TRE) of 4.2 mm on landmark centroids and a median Dice score of 0.88 on the prostate gland.
  • Speed: The trained model can perform more than 4 registrations per second on a single GPU, making it well-suited for real-time intraoperative use.
  • Superiority: The combined Global+Local approach significantly outperformed using either network in isolation, proving that the architectural constraint is vital for handling interventional data.

Limitations and Future Work

  • Label Dependency: While the system is "label-free" during surgery, it requires a massive amount of expert-labeled data for training. Generating these labels is labor-intensive.
  • Generalization: The current model was trained on a specific clinical trial dataset. The authors note that future work is needed to see how well this generalizes to data from different hospitals or different imaging hardware.
  • Structure Specificity: The registration is most accurate near the labels provided during training. If a critical structure wasn't labeled in the training set, the network might not align it as precisely.

Summary updated

Abstract

Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.
BibTeX
@inproceedings{hu2018label,
  author = {Yipeng Hu and Marc Modat and Eli Gibson and Nooshin Ghavami and Ester Bonmati and Caroline M. Moore and Mark Emberton and J. Alison Noble and Dean C. Barratt and Tom Vercauteren},
  title = {{Label-driven weakly-supervised learning for multimodal deformarle image registration}},
  booktitle = {ISBI},
  year = {2018},
  url = {https://openreview.net/forum?id=_tmvntzOr8h},
}

Loading related content...

Figure