Deep Learning for Medical Image Analysis

Mina Rezaei, Haojin Yang, Christoph MeinelarXiv 2017 · OpenReview

Abstract

This report describes research activities at the Hasso Plattner Institute, summarizing a Ph.D. plan and several novel, end-to-end trainable approaches for analyzing medical images using deep learning algorithms. As a primary application, we explore various methods for brain abnormality detection, recognition, and segmentation. This report was originally prepared for the doctoral consortium at the AIME-2017 conference.

Introduction: The Challenge of Brain MRI Analysis

Analyzing brain Magnetic Resonance Imaging (MRI) is a monumental task for radiologists. The brain’s structure is incredibly complex, and lesions vary significantly in size, shape, contrast, and location. Manual diagnosis is not only time-consuming but also prone to human subjectivity.

This research addresses the need for a robust, end-to-end trainable system that can assist clinicians by automatically identifying abnormalities. A central challenge is the multi-modal nature of medical imaging; a single patient may have T1-weighted, T2-weighted, and FLAIR scans, each revealing different tissue properties. Integrating these disparate data sources into a cohesive diagnostic tool is the primary hurdle addressed in this work.

Methods: A Multi-Tiered Diagnostic Pipeline

The proposed research utilizes a three-tiered approach to automate the diagnostic process:

  1. Classification: A deep Convolutional Neural Network (CNN) architecture, consisting of seven convolutional layers and three fully connected layers, is employed. Instead of a standard Softmax output, a 5-way Support Vector Machine (SVM) categorizes images into five classes: Healthy, Alzheimer’s, High-Grade Glioma (HGG), Low-Grade Glioma (LGG), and Multiple Sclerosis (MS).
  2. Detection and Localization: To localize lesions, a two-pathway architecture was developed. A "Global" path analyzes the whole image for context, while a "Local" path uses a fine-tuned VGG-16 and Fast R-CNN framework to extract features from specific patches. These paths are merged using Spatial Pyramid Pooling (SPP).
  3. Semantic Segmentation: For pixel-level precision, the Faster R-CNN framework is adapted to map exact lesion boundaries. This masking is critical for surgical planning and monitoring treatment response.
The proposed multi-stage deep learning pipeline, transitioning from initial classification to precise semantic segmentation and performance analysis.

Datasets

The research utilizes five distinct datasets to evaluate performance across different pathologies:

  • IXI Dataset: ~600 MR images of healthy subjects.
  • BRATS (2015/2016): The benchmark for Brain Tumor Segmentation (HGG and LGG).
  • OASIS: Focused on Alzheimer’s disease (416 subjects).
  • ISBI 2008: Used for Multiple Sclerosis lesion segmentation.
  • ISLES 2016: Focused on Ischemic Stroke Lesion Segmentation.

Key Insights: Multi-Modality and Context

  • The Power of Multi-Modality: Performance improves significantly when combining MRI sequences. For example, using only a T1 scan yielded a Dice score of ~61%, but integrating T1, T1c, T2, and FLAIR sequences increased the score to 92.44%.
  • Global + Local Context: The two-pathway approach mimics radiological workflows—evaluating the whole brain for context while zooming in on suspicious regions for detail. This combination significantly boosted the Dice coefficient for tumor detection.
  • GANs for Data Scarcity: Generative Adversarial Networks (GANs) are proposed to synthesize medical images, addressing the common bottleneck of limited labeled datasets in medical AI.

Experimental Results

The multi-stage pipeline demonstrated high performance across all diagnostic tasks:

  • Classification Accuracy: The 7-layer CNN achieved 95.07% accuracy across the five categories of brain images.
  • Tumor Detection: On the BRATS-2015 benchmark, the model achieved 94.3% accuracy for whole tumor detection with a sensitivity of 0.89.
  • Segmentation: The instance segmentation approach reached 89% accuracy for non-enhancing tumor segmentation.
  • Modality Impact: Multi-modal integration improved detection results by up to 20% in gliomas and 30% in ischemic stroke recognition.
Confusion matrix for the 5-way classification task, demonstrating high accuracy across healthy and pathological brain states.

Conclusion and Future Directions

While results are promising, several areas for future work remain. Transitioning from 2D slices to 3D semantic segmentation is critical to better capture the volumetric nature of brain lesions. Additionally, the use of GANs for cross-modality synthesis (e.g., generating CT from MRI) could reduce the need for multiple expensive scans. Finally, the research aims to implement data and model parallelism on GPUs to ensure the platform is fast enough for real-time clinical use.

Summary updated

Abstract

This report describes my research activities in the Hasso Plattner Institute and summarizes my Ph.D. plan and several novels, end-to-end trainable approaches for analyzing medical images using deep learning algorithm. In this report, as an example, we explore different novel methods based on deep learning for brain abnormality detection, recognition, and segmentation. This report prepared for the doctoral consortium in the AIME-2017 conference.
BibTeX
@misc{rezaei2017deep,
  author = {Mina Rezaei and Haojin Yang and Christoph Meinel},
  title = {{Deep Learning for Medical Image Analysis}},
  year = {2017},
  eprint = {1708.08987},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.08987},
}
Figure