Grayscale medical image segmentation method based on 2D&3D object detection with deep learning

Yunfei Ge, Qing Zhang, Yuantao Sun, Yidong Shen, Xijiong WangBmc Medical Imaging 2022 · DOI

TL;DR

This paper introduces a novel method for segmenting grayscale medical images by combining 2D and 3D object detection with deep learning. By transforming image pixels into a 3D point cloud, the approach leverages 3D object detection to identify regions of interest and determine segmentation thresholds, achieving high accuracy and outperforming existing methods on various medical imaging datasets.

What Problem Does This Paper Address?

Accurate segmentation of grayscale medical images is crucial for computer-aided diagnosis. Traditional methods like thresholding are computationally simple but often struggle with noise and variations in image intensity, requiring extensive pre-processing. Deep learning approaches, particularly encoder-decoder networks, offer high accuracy but demand large amounts of labeled training data, which is expensive and time-consuming to acquire in the medical domain. This paper aims to bridge this gap by developing a segmentation method that is both accurate and data-efficient, leveraging the strengths of object detection techniques.

Methods

The proposed method integrates 2D and 3D object detection with deep learning in a multi-stage pipeline:

  1. Interest Region Identification (2D Object Detection): A fine-tuned 2D object detection network (YOLOv3) is used to identify regions of interest (ROIs) containing potential segmentation objects. This step helps to reduce noise by focusing on relevant image areas. Transfer learning, specifically fine-tuning the last three layers of a pre-trained YOLOv3 model, is employed to adapt it to medical images with limited labeled data.
  2. 3D Point Cloud Generation: Pixels within the identified ROIs are converted into a 3D point cloud. The first two dimensions of the point cloud represent the pixel's spatial location (x, y), and the third dimension represents its grayscale value (z). This transforms the 2D image segmentation problem into a 3D object detection task.
  3. 3D Object Detection and Bounding Box Regression: A two-stage 3D object detection approach is applied to the point cloud.
  4. Proposal Generation: Anchor boxes are used to generate initial 3D bounding box proposals. A PointNet-based classifier is then employed to estimate the classification scores of points within these proposals. Non-Maximum Suppression (NMS) with 3D Intersection-over-Union (IoU) is used to refine these proposals into candidate bounding boxes.
  5. Refinement: A class-specific bounding box linear regression model is trained to refine the location and scale of the proposal bounding boxes, minimizing the errors between predicted and ground truth boxes.
  6. Segmentation Result Generation: The top and bottom boundaries of the refined 3D bounding boxes are interpreted as the thresholding values for segmentation. By projecting the points within these bounding boxes back into the 2D image, the segmented object is composed.

Datasets

The study utilizes several publicly available datasets for training and validation:

  • Musculoskeletal Radiographs (MURA): A large dataset of X-ray images of the upper and lower extremities.
  • Lower Extremity Radiographs (LERA): Another dataset from Stanford University focusing on lower extremity X-rays.
  • Chest Radiographs (CheXpert): A large dataset of chest X-ray images from Stanford University.
  • Phalanx and Forearm X-ray Images: Custom datasets acquired using a portable X-ray machine.

A total of 2509 cases from MURA and LERA, 3100 cases from CheXpert, and 500 phalanx and forearm X-ray images were used for training and validation.

Key Insights and Ideas

  • Bridging 2D and 3D for Segmentation: The core innovation lies in transforming a 2D image segmentation problem into a 3D object detection task by encoding pixel intensity as a third dimension in a point cloud. This allows for leveraging powerful 3D object detection techniques.
  • Data Efficiency through Transfer Learning: The method heavily relies on transfer learning for both 2D (YOLOv3) and 3D (PointNet) components. This is crucial for medical imaging where labeled data is scarce, enabling accurate segmentation with smaller datasets.
  • Interest Regions for Noise Reduction: Using 2D object detection to first identify ROIs effectively filters out irrelevant background noise, improving the focus and accuracy of subsequent 3D processing.
  • Grayscale Value as a Spatial Dimension: Representing grayscale values as the Z-axis in a 3D point cloud provides an intuitive way to group pixels with similar intensities, which is fundamental to grayscale image segmentation.

Key Results

The proposed method achieved impressive segmentation performance across the evaluated datasets:

  • IoU Scores: 0.92 for musculoskeletal radiographs, 0.88 for chest radiographs, and 0.94 for X-ray machine images.
  • DSC Scores: 0.96 for musculoskeletal radiographs, 0.93 for chest radiographs, and 0.94 for X-ray machine images.
  • Comparison: The method outperformed five state-of-the-art models (FCN, UNet, UNet++, Res-UNet, Attention UNet) in terms of IoU scores, demonstrating superior accuracy. For instance, on musculoskeletal radiographs, the proposed method achieved an IoU of 0.92, compared to 0.91 for Res-UNet and 0.90 for Attention UNet.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the method is effective, it relies on the availability of pre-trained models for both 2D and 3D object detection. The performance might be sensitive to the quality and domain of these pre-trained models. Additionally, the training process, while efficient due to transfer learning, still requires significant computational resources.
  • Future Work: The authors suggest that the proposed method can be flexibly adapted and upgraded by changing or improving the pre-trained models for both 2D and 3D object detection. They also propose that this approach can be extended to other types of grayscale medical images beyond bone and chest X-rays, highlighting its broad applicability.

Summary updated

BibTeX
@article{ge2022grayscale,
  author = {Yunfei Ge and Qing Zhang and Yuantao Sun and Yidong Shen and Xijiong Wang},
  title = {{Grayscale medical image segmentation method based on 2D\&3D object detection with deep learning}},
  journal = {Bmc Medical Imaging},
  year = {2022},
  doi = {10.1186/s12880-022-00760-2},
}
Figure