Montage based 3D medical image retrieval from traumatic brain injury cohort using deep convolutional neural network

Cailey I. Kerley, Yuankai Huo, Shikha Chaganti, Shunxing Bao, Mayur B. Patel, Bennett A. Landman

TL;DR

This paper introduces a novel method, dMIR, to automatically identify and retrieve whole brain CT scans from large clinical datasets. By converting 3D CT volumes into 2D "montage" images and using a deep convolutional neural network (ResNet-50), dMIR significantly reduces the time and resources needed for manual image filtering, achieving near-perfect accuracy.

What Problem Does This Paper Address?

Traumatic Brain Injury (TBI) research relies heavily on analyzing brain CT scans. However, clinical CT scans often capture the entire body, meaning only a small fraction (around 15%) of acquired scans actually contain usable whole brain images. Manually sifting through thousands of scans to find the relevant ones is a tedious, time-consuming, and expensive process, especially for large-scale studies. This bottleneck hinders efficient TBI research.

Methods

The core innovation is the deep montage-based image retrieval (dMIR) pipeline. Instead of directly processing the 3D CT volumes, dMIR transforms them into 2D representations called "montage images."

Here's how it works:

  1. Resampling and Padding: Each 3D CT scan is first resampled to a consistent in-plane resolution (512x512). If the scan has more than 36 slices, it's downsampled to 36 slices. If it has fewer than 36 slices, it's zero-padded to reach 36 slices. This ensures a uniform volume size of 512x512x36.
  2. Montage Image Generation: The 36 slices of the processed 3D volume are then concatenated along the z-direction to form a single 2D "montage" image. This montage image effectively captures the spatial information of the entire brain volume in a 2D format.
  3. Deep Convolutional Neural Network (DCNN) Classification: A pre-trained 50-layer ResNet (ResNet-50) is used to classify these montage images. The network is trained to distinguish between two classes: "usable whole brain CT scans" (class 1) and "unusable scans" (class 0). The network was trained from scratch with specific parameters like a batch size of 8, a learning rate of 0.0001, and an Adam optimizer. A weighted cross-entropy loss was used to handle class imbalance.

Datasets

The study utilized a dataset of 2000 clinically acquired TBI CT scans from Vanderbilt University Medical Center (VUMC). These scans were deidentified and used under Institutional Review Board approval.

  • Training Data: 794 scans from 80 subjects.
  • Validation Data: 206 scans from 21 subjects.
  • Testing Data: 1000 scans from 551 subjects.

The paper provides visual examples of both usable and unusable scans, categorizing unusable scans into those that don't include the brain at all, and those that include only parts of the brain or other anatomical regions (like neck or maxillofacial structures).

Key Insights and Ideas

  • Bridging 2D and 3D: The most significant insight is the clever transformation of a 3D medical image problem into a 2D image classification task. This allows leveraging the vast advancements and powerful architectures developed for 2D image analysis (like ResNet) for 3D medical data.
  • Montage as a Representation: The "montage image" is a novel way to represent the entirety of a 3D volume in a compact 2D format. This representation proves effective for capturing the essential characteristics needed for classification.
  • Efficiency Gain: By automating this retrieval process, the paper demonstrates a clear path to significantly reducing manual labor, saving time and resources in large-scale medical imaging studies.

Key Results

The dMIR pipeline achieved impressive results:

  • Validation Data: Accuracy, recall, precision, and F1-score of 1.0 for the 206 validation scans.
  • Testing Data: Accuracy of 0.988, recall of 0.962, precision of 0.962, and F1-score of 0.962 for the 1000 testing scans.
  • ROC Curves: The Receiver Operating Characteristic (ROC) curves show an Area Under the Curve (AUC) of 1.0 for validation and 0.9965 for external testing, indicating excellent discriminative power.
  • Error Analysis: The paper notes that 12 misclassifications occurred in the testing stage (six false positives and six false negatives), and provides visual examples of these failures.

Limitations and Future Work

  • Limitations: The authors acknowledge that while ResNet-50 performed exceptionally well, they did not compare it against other deep learning architectures for this specific task, as they had already achieved near-perfect results. The current study focused on TBI CT scans.
  • Future Work: The authors suggest that exploring other deep learning networks could be beneficial. They also propose that the dMIR method could be extended to other large-scale medical image archives for quality assurance purposes, potentially improving the management and usability of vast medical imaging datasets.

Summary updated

Abstract

Brain imaging analysis on clinically acquired computed tomography (CT) is essential for the diagnosis, risk prediction of progression, and treatment of the structural phenotypes of traumatic brain injury (TBI). However, in real clinical imaging scenarios, entire body CT images (e.g., neck, abdomen, chest, pelvis) are typically captured along with whole brain CT scans. For instance, in a typical sample of clinical TBI imaging cohort, only ~15% of CT scans actually contain whole brain CT images suitable for volumetric brain analyses; the remaining are partial brain or non-brain images. Therefore, a manual image retrieval process is typically required to isolate the whole brain CT scans from the entire cohort. However, the manual image retrieval is time and resource consuming and even more difficult for the larger cohorts. To alleviate the manual efforts, in this paper we propose an automated 3D medical image retrieval pipeline, called deep montage-based image retrieval (dMIR), which performs classification on 2D montage images via a deep convolutional neural network. The novelty of the proposed method for image processing is to characterize the medical image retrieval task based on the montage images. In a cohort of 2000 clinically acquired TBI scans, 794 scans were used as training data, 206 scans were used as validation data, and the remaining 1000 scans were used as testing data. The proposed achieved accuracy=1.0, recall=1.0, precision=1.0, f1=1.0 for validation data, while achieved accuracy=0.988, recall=0.962, precision=0.962, f1=0.962 for testing data. Thus, the proposed dMIR is able to perform accurate CT whole brain image retrieval from largescale clinical cohorts.
BibTeX
@inproceedings{kerley2019montage,
  author = {Cailey I. Kerley and Yuankai Huo and Shikha Chaganti and Shunxing Bao and Mayur B. Patel and Bennett A. Landman},
  title = {{Montage based 3D medical image retrieval from traumatic brain injury cohort using deep convolutional neural network}},
  booktitle = {Image Processing},
  year = {2019},
  doi = {10.1117/12.2512559},
}
Figure