Keypoint Transfer for Fast Whole-Body Segmentation

Christian Wachinger, Matthew Toews, Georg Langs, William M. Wells Iii, Polina GollandIEEE TMI 2018 · DOI · OpenReview

TL;DR

This paper introduces "Keypoint Transfer," a novel method for segmenting organs in medical images that significantly speeds up the process by using sparse keypoint correspondences instead of dense image registration. It achieves comparable accuracy to traditional multi-atlas methods but is orders of magnitude faster and requires no explicit training phase, making it highly scalable.

What Problem Does This Paper Address?

Accurate segmentation of organs in medical images is crucial for diagnosis and treatment planning. Traditional methods, like multi-atlas segmentation, often rely on computing dense correspondences (deforming an atlas image to match the target image). This process, especially for large images like whole-body scans, is computationally intensive and time-consuming. Furthermore, these methods typically require a dedicated training phase and struggle with images that have a variable field-of-view (e.g., scans focused on a specific region). This paper aims to address these limitations by developing a faster, more flexible segmentation approach.

Methods

The core of the Keypoint Transfer method lies in leveraging sparse correspondences between automatically detected "keypoints" in training and testing images. The algorithm proceeds in three main steps:

  1. Keypoint Matching: Distinctive image locations (keypoints) are automatically identified in both training and test images using a saliency function. These keypoints are then matched based on their descriptors (a 64-dimensional histogram of local image intensity gradients) and spatial constraints. A two-stage matching process is employed, first finding nearest neighbors and then refining matches with spatial constraints derived from a Hough transform to estimate global translation.
  2. Keypoint Voting: For each keypoint in the test image, an organ label is inferred by voting among the labels of its matched keypoints in the training images. The contribution of each match is weighted by the similarity of its descriptor and the reliability of the match itself. This results in a probabilistic label for each test keypoint.
  3. Probabilistic Transfer of Organ Segmentations: The inferred keypoint labels are then used to transfer entire organ segmentation masks from the training images to the test image. This transfer is modulated by the local intensity similarity between the training and test images, and the certainty of the keypoint label voting and match. This process generates probability maps for each organ, from which the final segmentation is derived.

A key novelty is the avoidance of dense registration and explicit training. The method is inherently robust to variations in field-of-view.

Datasets

The authors evaluated their method on the publicly available Visceral dataset. This dataset includes:

  • Gold Corpus: 20 images for each of four contrasts (ceCT, wbCT, ceMR, wbMR) with manual annotations.
  • Silver Corpus: A larger set of images (65 ceCT, 62 wbCT, 71 ceMR, 37 wbMR) with labels generated by fusing results from multiple segmentation methods.

Experiments were also conducted on cropped images to simulate limited field-of-view scans.

Key Insights and Ideas

  • Sparse is Sufficient: The paper demonstrates that using sparse keypoint correspondences, rather than dense image registration, is sufficient for accurate organ segmentation, leading to massive speedups.
  • No Training Phase Needed: The algorithm's ability to segment new images without an explicit training stage is a significant advantage, making it highly adaptable and scalable.
  • Robustness to Field-of-View: Keypoint transfer naturally handles variations in the field-of-view, a common challenge in clinical imaging, without requiring algorithm modifications.
  • Descriptor Power: The use of a 3D extension of the SIFT descriptor proves robust to intensity variations, enabling reliable matching across different imaging modalities and subjects.
  • Probabilistic Fusion: The probabilistic transfer and voting mechanism allows for a more robust segmentation by considering the certainty of matches and labels.

Key Results

  • Speed: Keypoint transfer is approximately three orders of magnitude faster than common multi-atlas segmentation methods. For example, segmenting ten organs on a ceCT scan takes about half a minute, compared to much longer times for multi-atlas approaches.
  • Accuracy: The segmentation accuracy, measured by Dice overlap, compares favorably to multi-atlas segmentation methods. In several cases, keypoint transfer achieved statistically significantly higher accuracy for specific organs (e.g., kidneys, aorta, spleen, trachea).
  • Limited Field-of-View: The method successfully segmented partial scans (e.g., kidneys and spleen) without any modifications, a task where multi-atlas registration failed due to difficulties in initial alignment.
  • Matching Constraints: The study found that all matching constraints (scale, spatial, and distance ratio) were important, with the spatial constraint showing the largest impact on accuracy when removed.

Limitations and Future Work

  • Small Organs/Low Texture: The paper acknowledges that the method might be limited for very small organs or those lacking salient texture, as keypoint identification might be challenging.
  • Neighboring Organ Transfer: While experiments with transferring segmentations from neighboring organs were explored, they did not consistently improve results for whole-body scans, possibly due to the large size of some organs.
  • MRI Intensity Inhomogeneity: For MRI scans, intensity variations can impact the accuracy of the weighting function. The authors suggest that approaches for correcting MRI intensity inhomogeneity could improve future results.
  • Future Directions: The authors suggest exploring the use of keypoints to support atlas selection schemes and investigating more sophisticated ways to transfer segmentations from neighboring organs, especially for cases where keypoints are distant from the target organ. They also note that the speed of keypoint transfer could allow for its use as an initial step for other, more complex segmentation approaches.

Summary updated

Abstract

We introduce an approach for image segmentation based on sparse correspondences between keypoints in testing and training images. Keypoints represent automatically identified distinctive image locations, where each keypoint correspondence suggests a transformation between images. We use these correspondences to transfer label maps of entire organs from the training images to the test image. The keypoint transfer algorithm includes three steps: (i) keypoint matching, (ii) voting-based keypoint labeling, and (iii) keypoint-based probabilistic transfer of organ segmentations. We report segmentation results for abdominal organs in whole-body CT and MRI, as well as in contrast-enhanced CT and MRI. Our method offers a speed-up of about three orders of magnitude in comparison to common multi-atlas segmentation, while achieving an accuracy that compares favorably. Moreover, keypoint transfer does not require the registration to an atlas or a training phase. Finally, the method allows for the segmentation of scans with highly variable field-of-view.
BibTeX
@article{wachinger2018keypoint,
  author = {Christian Wachinger and Matthew Toews and Georg Langs and William M. Wells III and Polina Golland},
  title = {{Keypoint Transfer for Fast Whole-Body Segmentation}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2018},
  doi = {10.1109/tmi.2018.2851194},
}
Figure