3D Aggregated Faster R-CNN for General Lesion Detection

Ning Zhang, Yu Cao, Benyuan Liu, Yan Luo

Abstract

Lesions are damages and abnormalities in tissues of the human body. Many of them can later turn into fatal diseases such as cancers. Detecting lesions are of great importance for early diagnosis and timely treatment. To this end, Computed Tomography (CT) scans often serve as the screening tool, allowing us to leverage the modern object detection techniques to detect the lesions. However, lesions in CT scans are often small and sparse. The local area of lesions can be very confusing, leading the region based classifier branch of Faster R-CNN easily fail. Therefore, most of the existing state-of-the-art solutions train two types of heterogeneous networks (multi-phase) separately for the candidate generation and the False Positive Reduction (FPR) purposes. In this paper, we enforce an end-to-end 3D Aggregated Faster R-CNN solution by stacking an "aggregated classifier branch" on the backbone of RPN. This classifier branch is equipped with Feature Aggregation and Local Magnification Layers to enhance the classifier branch. We demonstrate our model can achieve the state of the art performance on both LUNA16 and DeepLesion dataset. Especially, we achieve the best single-model FROC performance on LUNA16 with the inference time being 4.2s per processed scan.

Introduction

Detecting lesions, such as lung nodules, in Computed Tomography (CT) scans is a classic "needle in a haystack" problem. A typical chest CT scan covers a volume exceeding 20,000 $cm^3$, while a target lesion might be smaller than 5mm in diameter—a volume ratio of less than 1:100,000. Locally, these small lesions can appear nearly identical to cross-sections of blood vessels, making them difficult to distinguish from healthy tissue.

Standard object detection frameworks like Faster R-CNN often struggle in this domain because their classifier branches lack the resolution and context required to differentiate these "hard negatives." Consequently, most current state-of-the-art solutions rely on a multi-phase process: a shallow network for candidate generation followed by a separate, deep False Positive Reduction (FPR) network. While effective, this approach is computationally expensive and difficult to optimize end-to-end.

Methodology

The proposed 3D Aggregated Faster R-CNN integrates the FPR step directly into the detection architecture, creating a unified, end-to-end solution.

Architecture of the 3D Aggregated Faster R-CNN, featuring the 3D U-Net backbone, RPN branch, and the FPR branch with ROI Align.

Key architectural innovations include:

  • 3D U-Net Backbone with DenseNet Blocks: A 3D U-Net structure captures multi-scale features, while DenseNet blocks are utilized to enhance gradient flow and feature reuse.
  • Diameter Alignment (Feature Aggregation): Instead of extracting features from a single level of the feature pyramid, the model aggregates features across multiple levels. This provides small nodules with more surrounding context and large nodules with more internal detail.
  • Local Magnification: To manage the memory constraints of high-resolution 3D feature maps, the authors up-sample Region of Interest (RoI) crops locally before classification. This acts like a digital magnifying glass, allowing the network to inspect fine details without requiring a global increase in resolution.
  • Adaptive Focal Loss: To address extreme class imbalance (often 1 positive for every 10,000 negatives), the loss function dynamically shifts focus from easy negatives to hard positives during training.

Datasets and Evaluation

The model was evaluated on two primary benchmarks:

  1. LUNA16: A specialized dataset for pulmonary nodule detection containing 888 CT scans, serving as the standard for lung nodule detection.
  2. DeepLesion: A diverse NIH dataset containing over 32,000 lesions across various organs (liver, bone, kidney, etc.), testing the model's general detection capabilities.

Key Results and Discussion

The 3D Aggregated Faster R-CNN achieved state-of-the-art performance while maintaining high efficiency. On the LUNA16 benchmark, it reached a 0.939 FROC score, the highest reported for a single-model approach at the time of publication.

FROC curves comparing the proposed 3D-AG models against the baseline RPN, demonstrating significant sensitivity gains.

Key insights from the results include:

  • Joint Training Benefits: Simultaneous training of the RPN and FPR branches acted as a regularizer, improving the performance of both. The FPR branch imposes a scale-invariant constraint that helps the backbone learn more robust features.
  • Efficiency: The end-to-end architecture and local up-sampling enabled an inference time of 4.2 seconds per scan, which is 2-3 times faster than previous top-performing multi-phase pipelines.
  • Generalization: The model outperformed baseline RPNs on DeepLesion, showing particular strength in "Soft Tissue" lesions, though performance varied across different organ types like bone and kidney.

Conclusion

This work demonstrates that a unified 3D architecture can overcome the limitations of traditional multi-phase lesion detection systems. By combining context (Diameter Alignment) with fine-grained detail (Local Magnification), the 3D Aggregated Faster R-CNN provides a fast and accurate tool for clinical workflows. Future research may focus on integrating automated segmentation and exploring more interpretable features to further assist radiologists in diagnostic tasks.

Summary updated

Abstract

Lesions are damages and abnormalities in tissues of the human body. Many of them can later turn into fatal diseases such as cancers. Detecting lesions are of great importance for early diagnosis and timely treatment. To this end, Computed Tomography (CT) scans often serve as the screening tool, allowing us to leverage the modern object detection techniques to detect the lesions. However, lesions in CT scans are often small and sparse. The local area of lesions can be very confusing, leading the region based classifier branch of Faster R-CNN easily fail. Therefore, most of the existing state-of-the-art solutions train two types of heterogeneous networks (multi-phase) separately for the candidate generation and the False Positive Reduction (FPR) purposes. In this paper, we enforce an end-to-end 3D Aggregated Faster R-CNN solution by stacking an "aggregated classifier branch" on the backbone of RPN. This classifier branch is equipped with Feature Aggregation and Local Magnification Layers to enhance the classifier branch. We demonstrate our model can achieve the state of the art performance on both LUNA16 and DeepLesion dataset. Especially, we achieve the best single-model FROC performance on LUNA16 with the inference time being 4.2s per processed scan.
BibTeX
@misc{zhang2020d,
  author = {Ning Zhang and Yu Cao and Benyuan Liu and Yan Luo},
  title = {{3D Aggregated Faster R-CNN for General Lesion Detection}},
  year = {2020},
  eprint = {2001.11071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.11071},
}
Figure