Fine-Grained Lesion Annotation in CT Images With Knowledge Mined From Radiology Reports

Ke Yan, Yifan Peng, Zhiyong Lu, Ronald M. Summers

TL;DR

Researchers at the NIH have developed a way to automatically "read" and annotate CT scan lesions by mining existing radiology reports. By training a multi-scale convolutional neural network (CNN) on labels extracted directly from clinical text, their system can identify a lesion's body part, type, and attributes with an average AUC of 0.9083, effectively turning messy hospital records into a powerful training engine.

What Problem Does This Paper Address?

In a typical day, a radiologist identifies significant lesions in medical images and describes them in a report. However, building an AI to do this is notoriously difficult because of the "labeling bottleneck." Most existing AI models are hyper-specialized (e.g., only looking at liver cysts or skin cancer) because they require experts to manually label thousands of images—a process that is slow, expensive, and doesn't scale.

While radiology reports contain a wealth of information, the data is "noisy" and unstructured. There is often no direct, pixel-level link between a specific sentence in a report and a specific lesion in an image. This paper tackles the challenge of automatically mining these reports to create a fine-grained, large-scale dataset for "lesion annotation"—answering the fundamental question: "What is this thing I'm looking at?"

Methods

The authors proposed a two-part framework: a label mining pipeline and a specialized CNN architecture.

  1. Knowledge Mining: They used the RadLex lexicon to extract 145 fine-grained labels from radiology reports. To handle the complexity of medical language, they incorporated synonyms (e.g., "hepatic" = "liver") and used a hierarchical structure. If a report mentioned a "lung nodule," the system automatically added parent labels like "lung" and "chest" to the training data.
  2. Multi-Scale CNN: Not all features are created equal. To identify a lesion's texture (attribute), you need fine-grained, low-level image details. To identify its location (body part), you need the surrounding context. The authors modified a VGG-16 network to perform multi-scale feature fusion, using ROI (Region of Interest) pooling at different layers to capture both the lesion itself and its anatomical neighborhood.
  3. Noise-Robust Loss: Because radiology reports are often incomplete (a doctor might forget to mention a lesion is "spiculated"), the training labels are "noisy." The team implemented a soft bootstrapping loss function. This clever trick allows the model to rely partially on its own growing "intuition" during training, effectively smoothing out errors or omissions in the original text-mined labels.

Datasets

The study utilized the DeepLesion dataset, a massive collection of over 30,000 lesions from CT images released by the NIH.

  • Training Set: ~20,000 lesion-sentence pairs where hyperlinks in the reports pointed directly to specific image bookmarks.
  • Test Set: 1,872 lesions, which were manually verified to ensure the evaluation was grounded in ground truth.

Key Insights and Ideas

  • The Hierarchy Advantage: By using the parent-child relationships in RadLex, the researchers could "densify" their sparse data. A single specific mention provides multiple training signals across different levels of anatomical granularity.
  • Context vs. Detail: The "aha moment" in the architecture was realizing that lower-level CNN layers are better for attributes (like "calcified"), while higher-level layers are essential for body parts (like "mediastinum"). Combining these via multi-scale fusion was key to the high AUC.
  • Learning from "Silence": Radiologists don't describe every single feature of every lesion. The soft bootstrapping loss acknowledges this "silence" in the reports, preventing the model from being penalized too harshly when a label is missing but the image clearly shows the feature.

Key Results

The framework achieved an impressive average AUC of 0.9083 across all 145 labels.

  • Body Parts: Performed the best (e.g., Paraspinal: 0.9947, Lung: 0.9750), likely because anatomical context is visually distinct.
  • Lesion Types: Showed strong results for specific types like Ground-glass opacity (0.9770) and Liver mass (0.9690).
  • Attributes: These were the most challenging (e.g., Calcified: 0.7213, Large: 0.7189), as they are often subjective or subtle even for human readers.

The ablation study confirmed that adding the multi-scale structure and the noise-robust loss both provided significant jumps in accuracy over a standard VGG-16 baseline.

Limitations and Future Work

The authors acknowledge that their labels are still limited by the quality of the original radiology reports. If a radiologist doesn't mention an attribute, the model has a harder time learning it. Furthermore, some attributes are inherently subjective.

Future directions include: * Using more advanced Natural Language Processing (NLP) to further reduce noise in the mined reports. * Improving performance on "hard" and rare labels (some labels in the set had as few as 6 training examples). * Integrating this annotation tool into a fully automated CAD (Computer-Aided Diagnosis) system that first detects the lesion and then describes it.

Summary updated

Abstract

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and write sentences in the radiology report to describe them. In this paper, we study the lesion description or annotation problem as an important step of computer-aided diagnosis (CAD). Given a lesion image, our aim is to predict multiple relevant labels, such as the lesion's body part, type, and attributes. To address this problem, we define a set of 145 labels based on RadLex to describe a large variety of lesions in the DeepLesion dataset. We directly mine training labels from the lesion's corresponding sentence in the radiology report, which requires minimal manual effort and is easily generalizable to large data and label sets. A multi-label convolutional neural network is then proposed for images with multi-scale structure and a noise-robust loss. Quantitative and qualitative experiments demonstrate the effectiveness of the framework. The average area under ROC curve on 1,872 test lesions is 0.9083.
BibTeX
@inproceedings{yan2019fine,
  author = {Ke Yan and Yifan Peng and Zhiyong Lu and Ronald M. Summers},
  title = {{Fine-Grained Lesion Annotation in CT Images With Knowledge Mined From Radiology Reports}},
  booktitle = {ISBI},
  year = {2019},
  url = {https://openreview.net/forum?id=3k8TbzgHunw},
}

Loading related content...

Figure