End-to-end detection-segmentation network with ROI convolution

Zichen Zhang, Min Tang, Dana Cobzas, Dornoosh Zonoobi, Martin Jägersand, Jacob L. JaremkoISBI 2018 · OpenReview

TL;DR

This paper introduces a novel end-to-end neural network that combines object detection and segmentation for improved accuracy, particularly for small objects. By first localizing potential object regions and then using these regions to guide the segmentation process, the proposed "RPN-FCN" method outperforms existing approaches on a challenging clinical ultrasound dataset.

What Problem Does This Paper Address?

Accurate semantic segmentation, the task of classifying each pixel in an image, is crucial for many applications, especially in medical imaging. While Fully Convolutional Networks (FCNs) have been highly successful, they struggle with segmenting small objects. This is partly because standard FCNs treat all pixels equally, making it difficult to focus on tiny targets that are easily overwhelmed by background noise or larger structures. Humans, on the other hand, naturally perform a two-step process: first, they roughly locate an object of interest, and then they refine the segmentation within that localized region. This paper aims to bridge this gap by developing a neural network that mimics this human-like approach, integrating object detection to improve segmentation accuracy for challenging cases like small objects in noisy ultrasound images.

Methods

The core innovation is the RPN-FCN architecture, which integrates a Region Proposal Network (RPN), borrowed from object detection, with a segmentation network.

  • Backbone: The network uses a VGG16 as its feature extractor.
  • Localization Unit (RPN): This component, inspired by Faster R-CNN, identifies potential object regions (bounding boxes) and classifies them as foreground or background.
  • ROI Convolution: This is the key novel layer. Instead of using traditional ROI pooling (which can lose spatial detail by resizing features), ROI convolution directly applies convolutional filters to the feature maps within the proposed regions of interest (ROIs). This allows for image-wise computation, processing all ROIs simultaneously, and is designed to be differentiable for end-to-end training.
  • Segmentation Unit: The segmentation network takes the features processed by the ROI convolution layer and produces a segmentation map.
  • Loss Function: A multi-task loss is employed, combining bounding box regression, bounding box classification, and segmentation loss. Notably, the segmentation loss is defined on the full image, not just within ROIs, to avoid issues with overlapping predictions.

Datasets

The proposed method was evaluated on a clinical dataset of 3D ultrasound images of infant hips for the diagnosis of Developmental Dysplasia of the Hip (DDH). This dataset is characterized by:

  • Noisy ultrasound images.
  • Highly imbalanced classes, with the acetabulum (the target object) occupying only about 0.3% of the image area.
  • 49 3D ultrasound scans were used, split into 30 for training and 19 for testing.
  • 2D slices were used for evaluation, with most 3D volumes containing 256 slices.

Key Insights and Ideas

  • Mimicking Human Perception: The paper demonstrates that explicitly incorporating a localization step before segmentation can significantly improve performance, especially for small and challenging objects, by guiding the network's attention.
  • ROI Convolution for Efficiency and Accuracy: The novel ROI convolution layer offers a more efficient and potentially more accurate alternative to ROI pooling for integrating detection results into segmentation, as it avoids feature downsampling and allows for end-to-end training.
  • End-to-End Joint Learning: Training detection and segmentation jointly within a single network framework allows for better feature learning and mutual reinforcement between the two tasks.
  • Addressing Class Imbalance: While class imbalance is a major challenge, the proposed method, by focusing on localized regions, shows improved robustness compared to full-image segmentation methods.

Key Results

  • Improved Segmentation Accuracy: The RPN-FCN method achieved the best performance across all evaluated metrics (precision, recall, and Dice score) on the acetabulum segmentation task.
  • Quantitative Improvement: The paper reports a Dice score of 0.386 for RPN-FCN, significantly outperforming baselines like FCN-32s (0.223) and U-Net (0.049).
  • Qualitative Improvement: Visual results show that RPN-FCN produces smoother and more accurate segmentations, whereas U-Net often failed to predict any segmentation for the target object, and FCN-32s produced less refined boundaries.
  • Upsampling Strategy: By using ROI convolution and a higher resolution feature map, the network only needs to upsample by a factor of 16 (compared to 32 in FCN), which helps retain features of small objects.

Limitations and Future Work

  • Class Imbalance Handling: While the method shows improvement, the Dice scores are generally low, indicating that class imbalance remains a significant challenge. The authors note that class weighting in the loss function did not improve U-Net's performance in their experiments.
  • Hyperparameter Tuning: The authors mention that intensive hyperparameter tuning might further improve U-Net's performance.
  • Generalizability: The current evaluation is on a specific clinical task. Future work could explore the generalizability of the RPN-FCN architecture to other segmentation tasks, particularly those involving small objects or noisy data.
  • Deeper Backbones: The authors suggest that using deeper backbones like ResNet could further enhance accuracy.

Summary updated

Abstract

We propose an end-to-end neural network that improves the segmentation accuracy of fully convolutional networks by incorporating a localization unit. This network performs object localization first, which is then used as a cue to guide the training of the segmentation network. We test the proposed method on a segmentation task of small objects on a clinical dataset of ultrasound images. We show that by jointly learning for detection and segmentation, the proposed network is able to improve the segmentation accuracy compared to only learning for segmentation.
BibTeX
@inproceedings{zhang2018end,
  author = {Zichen Zhang and Min Tang and Dana Cobzas and Dornoosh Zonoobi and Martin Jägersand and Jacob L. Jaremko},
  title = {{End-to-end detection-segmentation network with ROI convolution}},
  booktitle = {ISBI},
  year = {2018},
  url = {https://openreview.net/forum?id=UkMTvrvm8XT},
}
Figure