3D Context Enhanced Region-based Convolutional Neural Network for End-to-End Lesion Detection

Ke Yan, Mohammadhadi Bagheri, Ronald M. Summers

Abstract

Detecting lesions from computed tomography (CT) scans is an important but difficult problem because non-lesions and true lesions can appear similar. 3D context is known to be helpful in this differentiation task. However, existing end-to-end detection frameworks of convolutional neural networks (CNNs) are mostly designed for 2D images. In this paper, we propose 3D context enhanced region-based CNN (3DCE) to incorporate 3D context information efficiently by aggregating feature maps of 2D images. 3DCE is easy to train and end-to-end in training and inference. A universal lesion detector is developed to detect all kinds of lesions in one algorithm using the DeepLesion dataset. Experimental results on this challenging task prove the effectiveness of 3DCE.

Introduction

In clinical radiology, detecting lesions in Computed Tomography (CT) scans is a "needle in a haystack" problem. A single 2D slice might show a small gray blob that looks like a lesion, but could easily be a blood vessel or a normal organ structure. To tell the difference, radiologists look at the "3D context"—how that blob changes as you scroll through adjacent slices.

While 3D Convolutional Neural Networks (CNNs) seem like the obvious tool for this, they face two major hurdles: computational cost and data scarcity. 3D convolutions are incredibly memory-intensive, often making it impossible to fit high-resolution volumetric data into standard GPU memory. Furthermore, most 3D models must be trained from scratch due to the lack of a "3D ImageNet," which often leads to overfitting on smaller medical datasets. The 3D Context Enhanced Region-based CNN (3DCE) provides a middle ground: a way to give 2D networks 3D "vision" without the overhead of full 3D convolutions.

The 3DCE Framework

The 3DCE architecture builds upon the Region-based Fully Convolutional Network (R-FCN) to bring 3D spatial awareness to 2D medical imaging detectors. By aggregating feature maps from multiple CT slices using a shared 2D backbone, it achieves state-of-the-art lesion detection while remaining memory-efficient.

Overview of the 3DCE architecture, showing the flow from multi-channel images to concatenated feature maps and the R-FCN head.

Feature-Level Fusion and Shared Weights

Instead of stacking multiple CT slices as input channels (data-level fusion), 3DCE processes neighboring slices through a shared 2D backbone (VGG-16). It then extracts feature maps for each slice and concatenates them. This allows the network to learn 2D features first and then "reason" about their 3D relationship in the feature space. The 2D backbone (Conv1–6) uses shared weights across all input slices, which significantly reduces the number of parameters compared to a 3D CNN, making the model easier to train and less prone to overfitting.

Modified R-FCN Head

The authors improved the standard R-FCN by adding a 2048D fully-connected layer and a ReLU layer after the Position-Sensitive ROI (PSROI) pooling. This extra "thinking power" boosted detection accuracy by about 0.7% with negligible speed loss. Unlike older methods that required a separate "False Positive Reduction" stage, 3DCE is trained end-to-end, making the pipeline simpler and faster.

Experimental Setup and Dataset

The study utilizes the DeepLesion dataset, a large-scale, diverse benchmark released by the NIH. It contains over 32,000 2D annotations of lesions from 10,594 CT studies. Unlike specialized datasets which might only contain lung nodules, DeepLesion includes kidney masses, bone lesions, liver tumors, enlarged lymph nodes, and more. For processing, CT intensities were normalized to a [0, 255] range using a wide window (-1024 to 3071 HU) to capture bone, soft tissue, and lung details simultaneously.

Performance Evaluation

3DCE consistently outperformed 2D-only baselines and standard data-fusion techniques. The researchers tested using 3, 9, 15, 21, and 27 slices of context. They found that while more context generally helps, the gains start to plateau, suggesting that a range of ~18–54mm of anatomical context is the "sweet spot" for most lesions.

FROC curves comparing 3DCE with different slice counts against 2D baselines and data-level fusion methods.

At a rate of 4 false positives per image, 3DCE (using 27 slices) achieved 84.37% sensitivity, compared to 80.32% for a standard 3-slice Faster R-CNN baseline. On a re-annotated "all-lesion" test set, 3DCE maintained a high sensitivity of 82.75%. Despite processing multiple slices, the 9-slice version of 3DCE runs in just 56ms per image, making it viable for real-time clinical applications.

Discussion and Future Directions

One primary limitation identified is slice interval sensitivity; the model performs better on scans with thinner slices (e.g., 1mm vs 5mm). When slices are too far apart, the "3D context" becomes disjointed, making feature aggregation less effective. Additionally, very small or very scattered lesions still pose a challenge for the Region Proposal Network (RPN).

Future work may involve extending 3DCE to other architectures (like ResNet or Mask R-CNN) and applying it to other volumetric tasks beyond detection, such as 3D segmentation or lesion growth tracking. By factorizing complex 3D filters into 2D spatial filters followed by 1D depth-wise aggregation, 3DCE proves that medical AI can leverage ImageNet-pretrained weights while still learning vital 3D spatial relationships.

Summary updated

Abstract

Detecting lesions from computed tomography (CT) scans is an important but difficult problem because non-lesions and true lesions can appear similar. 3D context is known to be helpful in this differentiation task. However, existing end-to-end detection frameworks of convolutional neural networks (CNNs) are mostly designed for 2D images. In this paper, we propose 3D context enhanced region-based CNN (3DCE) to incorporate 3D context information efficiently by aggregating feature maps of 2D images. 3DCE is easy to train and end-to-end in training and inference. A universal lesion detector is developed to detect all kinds of lesions in one algorithm using the DeepLesion dataset. Experimental results on this challenging task prove the effectiveness of 3DCE. We have released the code of 3DCE in https://github.com/rsummers11/CADLab/tree/master/lesion_detector_3DCE.
BibTeX
@inproceedings{yan2018d,
  author = {Ke Yan and Mohammadhadi Bagheri and Ronald M. Summers},
  title = {{3D Context Enhanced Region-based Convolutional Neural Network for End-to-End Lesion Detection}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  url = {https://openreview.net/forum?id=EFRWFsg8HKT},
}
Figure