ULDor: A Universal Lesion Detector for CT Scans with Pseudo Masks and Hard Negative Example Mining

Youbao Tang, Ke Yan, Yuxing Tang, Jiamin Liu, Jing Xiao, Ronald M. SummersISBI 2019 · OpenReview

TL;DR

This paper introduces ULDor, a universal lesion detector for CT scans that tackles the challenge of detecting diverse lesion types across the entire body. It achieves this by leveraging Mask R-CNN with novel "pseudo masks" for training and a "hard negative example mining" strategy to significantly reduce false positives, leading to improved detection accuracy.

What Problem Does This Paper Address?

Detecting lesions in CT scans is crucial for medical diagnosis, but it's a tough problem. Lesions can look very similar to normal tissues in terms of texture and intensity, making it hard to build a detector that works for all types of lesions across the whole body. Previous efforts often focused on specific lesion types (like lung nodules). A major hurdle is the immense effort required to manually annotate pixel-level masks for large datasets, which is essential for training advanced models like Mask R-CNN. This paper aims to overcome these challenges by creating a universal detector that doesn't rely on perfect manual annotations and can effectively distinguish true lesions from false alarms.

Methods

The core of ULDor is the Mask R-CNN architecture, a state-of-the-art object detection framework that includes a branch for predicting segmentation masks. The paper introduces two key innovations:

  • Pseudo Mask Construction: Since manually creating pixel-level masks for a large dataset is impractical, the authors generate "pseudo masks." These are derived from existing RECIST (Response Evaluation Criteria In Solid Tumors) annotations, which provide lesion diameters. By fitting ellipses to the RECIST endpoints, they create surrogate masks that are sufficient for training Mask R-CNN.
  • Hard Negative Example Mining (HNEM): To reduce false positives, which can lower the confidence scores of true positives, the paper proposes an iterative HNEM strategy. After an initial detection pass, the model identifies potential false positives that have high confidence scores but low segmentation overlap with the pseudo masks. These "hard negative examples" are then used to retrain the Mask R-CNN model, teaching it to better differentiate true lesions from these challenging false alarms.

The training process is iterative: Mask R-CNN is trained, then HNEM is applied to mine hard negatives, and the model is retrained with these new examples. This cycle continues until the validation loss is minimized.

Datasets

  • NIH DeepLesion dataset: This is the primary dataset used for both training and evaluation. It's a large-scale dataset containing 32,735 CT lesion images annotated with RECIST diameters, covering a wide variety of lesions across the entire body. The dataset is split into training (70%), validation (15%), and testing (15%) sets.

Key Insights and Ideas

  • Pseudo masks are a viable substitute for real masks: The paper demonstrates that even imperfect, generated "pseudo masks" can be effectively used to train a powerful segmentation-based detector like Mask R-CNN, bypassing the need for laborious manual annotation.
  • Hard negative mining is crucial for reducing false positives: Simply detecting lesions isn't enough; reducing false alarms is critical for clinical utility. HNEM proves to be a highly effective strategy for this, allowing the model to learn more discriminative features.
  • Iterative training is key: The combination of Mask R-CNN with pseudo masks and HNEM, trained iteratively, leads to a significant performance boost compared to using either component alone or a standard Mask R-CNN.
  • Universal detection is achievable: The approach successfully demonstrates the potential for a single model to detect a wide range of lesion types across different anatomical regions, moving beyond specialized detectors.

Key Results

  • The proposed ULDor method (Mask R-CNN + HNEM) achieved a sensitivity of 86.21% with an average of five false positives per image on the NIH DeepLesion test set.
  • This represents an absolute improvement of 5.11% over the previous state-of-the-art reported in [4] (which achieved 81.1% sensitivity at a similar false positive rate).
  • Qualitative comparisons showed that the Mask R-CNN model with pseudo masks performed better than without, and the addition of HNEM significantly reduced false positives compared to the model without it.
  • When considering more false positives as potential true lesions (Mask R-CNN + HNEM + FP), the method achieved a maximum sensitivity of 94.2%, outperforming the reference [4] (88.0%) at higher false positive rates.

Limitations and Future Work

  • Performance at low false positive rates: The paper notes that their method performs slightly worse than the reference [4] when the average false positive rate is less than 1. They attribute this to the backbone network (ResNet-101 vs. VGGNet-16 in [4]), suggesting that a larger backbone might learn more representative features for detecting top positives at very low FP rates.
  • Missed annotations: The authors acknowledge that there are some true lesions that might be missing annotations in the mined negative examples. This can lead the model to misclassify some true lesions as false positives during testing.
  • Similar appearances: The ULDor still struggles to detect lesions that have extremely similar appearances to their surrounding tissues, as shown in a failure example. This indicates room for further improvement in feature representation.
  • Future directions: The authors suggest that their method has the potential to be integrated with other tools (like lesion segmentation) to create fully automated clinical workflows. Further research could explore more robust methods for handling extremely similar lesion-tissue appearances and potentially investigate larger backbone architectures for improved performance at very low false positive rates.

Summary updated

Abstract

Automatic lesion detection from computed tomography (CT) scans is an important task in medical imaging analysis. It is still very challenging due to similar appearances (e.g. intensity and texture) between lesions and other tissues, making it especially difficult to develop a universal lesion detector. Instead of developing a specific-type lesion detector, this work builds a Universal Lesion Detector (ULDor) based on Mask R-CNN, which is able to detect all different kinds of lesions from whole body parts. As a state-of-the-art object detector, Mask R-CNN adds a branch for predicting segmentation masks on each Region of Interest (RoI) to improve the detection performance. However, it is almost impossible to manually annotate a large-scale dataset with pixel-level lesion masks to train the Mask R-CNN for lesion detection. To address this problem, this work constructs a pseudo mask for each lesion region that can be considered as a surrogate of the real mask, based on which the Mask R-CNN is employed for lesion detection. On the other hand, this work proposes a hard negative example mining strategy to reduce the false positives for improving the detection performance. Experimental results on the NIH DeepLesion dataset demonstrate that the ULDor is enhanced using pseudo masks and the proposed hard negative example mining strategy and achieves a sensitivity of 86.21% with five false positives per image.
BibTeX
@inproceedings{tang2019uldor,
  author = {Youbao Tang and Ke Yan and Yuxing Tang and Jiamin Liu and Jing Xiao and Ronald M. Summers},
  title = {{ULDor: A Universal Lesion Detector for CT Scans with Pseudo Masks and Hard Negative Example Mining}},
  booktitle = {ISBI},
  year = {2019},
  url = {https://openreview.net/forum?id=0Kqhfe0AV0},
}
Figure