Segmentation-by-detection: A cascade network for volumetric medical image segmentation

Min Tang, Zichen Zhang, Dana Cobzas, Martin Jägersand, Jacob L. Jaremko

TL;DR

The authors propose a "segmentation-by-detection" framework that uses a two-stage cascade to process 3D medical images. By first using a 2D Region Proposal Network (RPN) to identify areas of interest, the system creates an "attention mask" that guides a 3D U-Net to segment complex structures more accurately and efficiently than standard models.

What Problem Does This Paper Address?

Volumetric medical imaging (like MRI, CT, or 3D ultrasound) presents a "needle in a haystack" problem for deep learning. These volumes are often massive, computationally expensive to process in 3D, and frequently plagued by low signal-to-noise ratios (SNR).

Standard approaches usually fall into two camps:

  1. Slice-by-slice 2D analysis: This is computationally cheap and allows researchers to use models pre-trained on natural images, but it ignores the vital anatomical context between slices.
  2. Full 3D volumes: While these capture spatial context, they are incredibly memory-intensive and often struggle to converge because the network is "distracted" by the surrounding noise and irrelevant tissues in the large volume.

The authors aim to bridge this gap by focusing the network's "attention" only on the regions that actually matter.

Methods

The proposed framework is a cascade network consisting of two main modules:

  1. Detection Module (2D): Based on the Region Proposal Network (RPN) from the Faster R-CNN architecture, this module uses a VGG-16 backbone. It processes 2D slices to identify candidate bounding boxes (Regions of Interest, or RoIs) where the target anatomy likely resides.
  2. Attention Model: The 2D detections are stacked and concatenated to form a 3D binary mask. This mask acts as a spatial attention map.
  3. Segmentation Module (3D): A 3D U-Net takes two inputs: the original 3D ultrasound volume and the 3D attention mask. By having the mask as an additional channel, the 3D U-Net "distills" information from the potential region, effectively ignoring the surrounding noise.

This approach is novel because it uses 2D detection—a task that is easier to train and can leverage pre-trained weights—to simplify a complex 3D segmentation task.

Datasets

The model was evaluated on a specific clinical task: segmenting the femoral head in 3D ultrasound (3DUS) scans of infants. This is a critical step for diagnosing Developmental Dysplasia of the Hip (DDH).

  • Data Source: 19 3DUS scans from infants aged 4 to 111 days.
  • Preprocessing: The volumes were down-sampled to $128 \times 184 \times 96$ for the experiments.
  • Split: 10 scans were used for training (5 left, 5 right femoral heads), and 9 scans were reserved for testing.

Key Insights and Ideas

  • Localization is Easier than Segmentation: The authors argue that finding where an object is (detection) is a simpler task for a neural network than defining its exact boundaries (segmentation). By solving the easier problem first, they provide a "hint" that makes the harder problem more manageable.
  • Noise Reduction via Attention: In 3D ultrasound, the signal-to-noise ratio is often very low. By providing an attention mask, the segmentation module doesn't have to learn how to ignore the entire volume's noise; it only needs to focus on the refined area provided by the detector.
  • Hybrid 2D/3D Pipeline: This architecture cleverly uses 2D networks where they are most effective (leveraging ImageNet pre-trained weights for feature extraction) and 3D networks where they are necessary (capturing volumetric context).

Key Results

The "Segmentation-by-Detection" method significantly outperformed the standard 3D U-Net across all metrics:

  • Accuracy: The proposed method achieved an Average IoU (Intersection over Union) of 0.709, compared to 0.622 for the standard 3D U-Net—an 8.7% improvement.
  • Robustness: Even in the "worst-case" scenarios, the proposed method maintained an IoU of 0.512, whereas the standard 3D U-Net dropped to 0.483.
  • Convergence: Training loss curves showed that the cascade network converged faster and more stably than the standalone 3D U-Net.
  • Qualitative Quality: Visual results showed that the attention-guided model produced much smoother and more anatomically plausible contours compared to the "disrupted" results of the baseline.

Limitations and Future Work

The primary limitation is the dataset size; 19 scans is a very small sample for deep learning, though common in specialized medical imaging pilot studies. The authors also noted that the performance of the segmentation is heavily dependent on the quality of the detection module.

For future work, the authors suggest that the detection module could potentially be replaced or supplemented by a human-in-the-loop interactive tool. A human could provide a rough region of interest, and the 3D segmentation module would then provide the precise, refined boundaries. This could lead to even higher accuracy in clinical settings.

Summary updated

Abstract

We propose an attention mechanism for 3D medical image segmentation. The method, named segmentation-by-detection, is a cascade of a detection module followed by a segmentation module. The detection module enables a region of interest to come to attention and produces a set of object region candidates which are further used as an attention model. Rather than dealing with the entire volume, the segmentation module distills the information from the potential region. This scheme is an efficient solution for volumetric data as it reduces the influence of the surrounding noise. This is especially important for medical data with low signal-to-noise ratio. Experimental results on 3D ultrasound data of the femoral head shows superiority of the proposed method when compared with a standard fully convolutional network like the U-Net.
BibTeX
@inproceedings{tang2018segmentation,
  author = {Min Tang and Zichen Zhang and Dana Cobzas and Martin Jägersand and Jacob L. Jaremko},
  title = {{Segmentation-by-detection: A cascade network for volumetric medical image segmentation}},
  booktitle = {ISBI},
  year = {2018},
  url = {https://openreview.net/forum?id=2ZTNYRMBZKh},
}

Loading related content...

Figure