Fine-grained lesion annotation in CT images with knowledge mined from radiology reports

Ke Yan, Yifan Peng, Zhiyong Lu, Ronald M. Summers

Abstract

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and write sentences in the radiology report to describe them. In this paper, we study the lesion description or annotation problem as an important step of computer-aided diagnosis (CAD). Given a lesion image, our aim is to predict multiple relevant labels, such as the lesion's body part, type, and attributes. To address this problem, we define a set of 145 labels based on RadLex to describe a large variety of lesions in the DeepLesion dataset. We directly mine training labels from the lesion's corresponding sentence in the radiology report, which requires minimal manual effort and is easily generalizable to large data and label sets. A multi-label convolutional neural network is then proposed for images with multi-scale structure and a noise-robust loss. Quantitative and qualitative experiments demonstrate the effectiveness of the framework. The average area under ROC curve on 1,872 test lesions is 0.9083.

Introduction

Radiologists spend a significant portion of their day identifying and describing lesions in medical images. While these descriptions are captured in radiology reports, translating this clinical knowledge into automated systems is hindered by the "labeling bottleneck." Traditional computer-aided diagnosis (CAD) models are often hyper-specialized, requiring thousands of manually labeled images for specific conditions like liver cysts. This manual process is expensive and scales poorly.

Radiology reports contain a wealth of information, but the data is unstructured and "noisy." There is rarely a direct, pixel-level link between a sentence in a report and a specific lesion in an image. This research addresses the challenge of automatically mining these reports to create a fine-grained, large-scale dataset for lesion annotation, effectively answering the question: "What is this lesion?"

Methodology

The proposed framework consists of a label mining pipeline and a specialized multi-label convolutional neural network (CNN).

Knowledge Mining and Label Hierarchy

The researchers utilized the RadLex lexicon to extract 145 fine-grained labels from radiology reports. To manage medical language complexity, the system incorporates synonyms (e.g., "hepatic" and "liver") and a hierarchical structure. For instance, if a report mentions a "lung nodule," the system automatically infers parent labels like "lung" and "chest." This hierarchical approach "densifies" sparse data, providing multiple training signals from a single mention.

Label hierarchy and mining from radiology reports.

Multi-Scale CNN Architecture

Identifying a lesion requires both local texture details (for attributes like "calcified") and global anatomical context (for body parts like "mediastinum"). The authors modified a VGG-16 network to perform multi-scale feature fusion. By using ROI (Region of Interest) pooling at different layers (from Conv1_2 to Conv5_3), the model captures both the lesion's fine details and its surrounding environment.

Multi-scale CNN architecture for lesion annotation.

Noise-Robust Training

Radiology reports are often incomplete; a radiologist might not mention every attribute of a lesion. To handle this "noisy" data, the team implemented a soft bootstrapping loss function. This allows the model to partially rely on its own predictions during training, smoothing out omissions in the text-mined labels.

Dataset

The study used the DeepLesion dataset, containing over 30,000 lesions. The training set included approximately 20,000 lesion-sentence pairs linked via report bookmarks. A test set of 1,872 lesions was manually verified to ensure ground-truth accuracy.

Results

The framework achieved an average Area Under the ROC Curve (AUC) of 0.9083 across 145 labels.

  • Body Parts: Performance was highest in this category (e.g., Paraspinal: 0.9947, Lung: 0.9750), as anatomical context is visually distinct.
  • Lesion Types: Strong results were seen for types like Ground-glass opacity (0.9770) and Liver mass (0.9690).
  • Attributes: These proved most challenging (e.g., Calcified: 0.7213), likely due to their subjective nature and subtle visual cues.

Ablation studies confirmed that both the multi-scale structure and the noise-robust loss significantly improved accuracy over a standard VGG-16 baseline.

Discussion

While successful, the model's performance is inherently tied to the quality of the source radiology reports. If a feature is not mentioned by the radiologist, the model's learning signal is weakened. Future work will focus on using advanced Natural Language Processing (NLP) to further reduce noise in mined reports and improving performance on rare labels. Ultimately, this annotation tool could be integrated into fully automated CAD systems that both detect and describe lesions.

Summary updated

Abstract

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and write sentences in the radiology report to describe them. In this paper, we study the lesion description or annotation problem as an important step of computer-aided diagnosis (CAD). Given a lesion image, our aim is to predict multiple relevant labels, such as the lesion's body part, type, and attributes. To address this problem, we define a set of 145 labels based on RadLex to describe a large variety of lesions in the DeepLesion dataset. We directly mine training labels from the lesion's corresponding sentence in the radiology report, which requires minimal manual effort and is easily generalizable to large data and label sets. A multi-label convolutional neural network is then proposed for images with multi-scale structure and a noise-robust loss. Quantitative and qualitative experiments demonstrate the effectiveness of the framework. The average area under ROC curve on 1,872 test lesions is 0.9083.
BibTeX
@inproceedings{yan2019fine,
  author = {Ke Yan and Yifan Peng and Zhiyong Lu and Ronald M. Summers},
  title = {{Fine-grained lesion annotation in CT images with knowledge mined from radiology reports}},
  booktitle = {ISBI},
  year = {2019},
  url = {https://openreview.net/forum?id=3k8TbzgHunw},
}
Figure