Abstract
Detecting lesions from computed tomography (CT) scans is an important but difficult problem because non-lesions and true lesions can appear similar. 3D context is known to be helpful in this differentiation task. However, existing end-to-end detection frameworks of convolutional neural networks (CNNs) are mostly designed for 2D images. In this paper, we propose 3D context enhanced region-based CNN (3DCE) to incorporate 3D context information efficiently by aggregating feature maps of 2D images. 3DCE is easy to train and end-to-end in training and inference. A universal lesion detector is developed to detect all kinds of lesions in one algorithm using the DeepLesion dataset. Experimental results on this challenging task prove the effectiveness of 3DCE.
Introduction
In clinical radiology, detecting lesions in Computed Tomography (CT) scans is a "needle in a haystack" problem. A single 2D slice might show a small gray blob that looks like a lesion, but could easily be a blood vessel or a normal organ structure. To tell the difference, radiologists look at the "3D context"—how that blob changes as you scroll through adjacent slices.
While 3D Convolutional Neural Networks (CNNs) seem like the obvious tool for this, they face two major hurdles: computational cost and data scarcity. 3D convolutions are incredibly memory-intensive, often making it impossible to fit high-resolution volumetric data into standard GPU memory. Furthermore, most 3D models must be trained from scratch due to the lack of a "3D ImageNet," which often leads to overfitting on smaller medical datasets. The 3D Context Enhanced Region-based CNN (3DCE) provides a middle ground: a way to give 2D networks 3D "vision" without the overhead of full 3D convolutions.
The 3DCE Framework
The 3DCE architecture builds upon the Region-based Fully Convolutional Network (R-FCN) to bring 3D spatial awareness to 2D medical imaging detectors. By aggregating feature maps from multiple CT slices using a shared 2D backbone, it achieves state-of-the-art lesion detection while remaining memory-efficient.
Feature-Level Fusion and Shared Weights
Instead of stacking multiple CT slices as input channels (data-level fusion), 3DCE processes neighboring slices through a shared 2D backbone (VGG-16). It then extracts feature maps for each slice and concatenates them. This allows the network to learn 2D features first and then "reason" about their 3D relationship in the feature space. The 2D backbone (Conv1–6) uses shared weights across all input slices, which significantly reduces the number of parameters compared to a 3D CNN, making the model easier to train and less prone to overfitting.
Modified R-FCN Head
The authors improved the standard R-FCN by adding a 2048D fully-connected layer and a ReLU layer after the Position-Sensitive ROI (PSROI) pooling. This extra "thinking power" boosted detection accuracy by about 0.7% with negligible speed loss. Unlike older methods that required a separate "False Positive Reduction" stage, 3DCE is trained end-to-end, making the pipeline simpler and faster.
Experimental Setup and Dataset
The study utilizes the DeepLesion dataset, a large-scale, diverse benchmark released by the NIH. It contains over 32,000 2D annotations of lesions from 10,594 CT studies. Unlike specialized datasets which might only contain lung nodules, DeepLesion includes kidney masses, bone lesions, liver tumors, enlarged lymph nodes, and more. For processing, CT intensities were normalized to a [0, 255] range using a wide window (-1024 to 3071 HU) to capture bone, soft tissue, and lung details simultaneously.
Performance Evaluation
3DCE consistently outperformed 2D-only baselines and standard data-fusion techniques. The researchers tested using 3, 9, 15, 21, and 27 slices of context. They found that while more context generally helps, the gains start to plateau, suggesting that a range of ~18–54mm of anatomical context is the "sweet spot" for most lesions.
At a rate of 4 false positives per image, 3DCE (using 27 slices) achieved 84.37% sensitivity, compared to 80.32% for a standard 3-slice Faster R-CNN baseline. On a re-annotated "all-lesion" test set, 3DCE maintained a high sensitivity of 82.75%. Despite processing multiple slices, the 9-slice version of 3DCE runs in just 56ms per image, making it viable for real-time clinical applications.
Discussion and Future Directions
One primary limitation identified is slice interval sensitivity; the model performs better on scans with thinner slices (e.g., 1mm vs 5mm). When slices are too far apart, the "3D context" becomes disjointed, making feature aggregation less effective. Additionally, very small or very scattered lesions still pose a challenge for the Region Proposal Network (RPN).
Future work may involve extending 3DCE to other architectures (like ResNet or Mask R-CNN) and applying it to other volumetric tasks beyond detection, such as 3D segmentation or lesion growth tracking. By factorizing complex 3D filters into 2D spatial filters followed by 1D depth-wise aggregation, 3DCE proves that medical AI can leverage ImageNet-pretrained weights while still learning vital 3D spatial relationships.
Summary updated