Image Analysis in Digital Pathology Utilizing Machine Learning and Deep Neural Networks

Paris Panagiotis Amerikanos, Ilias MaglogiannisJournal of Personalized Medicine 2022 · DOI

TL;DR

This paper explores the application of deep learning (DL) frameworks, specifically Detectron2, for automated image analysis in digital pathology. By training on two distinct datasets, the authors demonstrate the potential of DL to enhance the accuracy and efficiency of tasks like mitosis detection and instance segmentation, aiming to assist pathologists and accelerate diagnoses.

What Problem Does This Paper Address?

Pathology, particularly the analysis of whole slide images (WSIs), is a critical but often subjective and time-consuming process. Manually identifying and quantifying features like mitotic figures (indicating cell division and tumor proliferation) is labor-intensive and prone to inter-observer variability. This paper addresses the need for automated, accurate, and efficient tools to aid pathologists in these tasks, thereby improving diagnostic speed and reliability. The challenge lies in developing DL models that can generalize well across different datasets and tasks within digital pathology.

Methods

The core of this research utilizes Detectron2, an open-source deep learning framework designed for object detection and instance segmentation. The authors leverage a Mask R-CNN architecture with a ResNet R50-FPN backbone, pre-trained on the COCO dataset. This approach allows for both identifying objects (like mitotic figures) and segmenting their precise boundaries.

Key methodological aspects include:

  • Transfer Learning: Using a model pre-trained on a large, general-purpose dataset (COCO) and fine-tuning it for specific medical imaging tasks.
  • Hyperparameter Tuning: A manual search approach was employed to optimize parameters like learning rate, batch size, and iterations to achieve the best performance on validation sets.
  • Data Preprocessing: This involved several steps tailored to each dataset, including:
  • Subimage Splitting: Large WSIs were divided into smaller, fixed-size patches (224x224 px) to match the CNN input requirements and avoid information loss from resizing.
  • Annotation Formatting: Ground truth annotations were converted into formats compatible with Detectron2 (e.g., COCO RLE).
  • Data Augmentation (Limited): While not extensively used in the initial experiments, the paper acknowledges its potential for improving robustness.
  • Staining Normalization (Explored and Rejected): An attempt was made to normalize image histograms, but it distorted visual features and reduced accuracy, leading to its rejection.

Datasets

The study utilized two publicly available datasets:

  • TUPAC16: This dataset contains 73 WSIs of breast cancer cases, annotated with the central points of mitotic figures. It was used for the object detection task, specifically identifying mitotic figures.
  • JPATHOL: This dataset comprises three sub-datasets for instance segmentation tasks:
  • Nuclei Segmentation: ROIs of ER+ breast cancer, containing annotated nuclei.
  • Epithelium Segmentation: ROIs of ER+ breast cancer, with annotations for epithelium regions.
  • Tubule Segmentation: ROIs from colorectal cancer, with annotated tubules.

Key Insights and Ideas

  • Unified Framework Potential: The research highlights the feasibility of using a single, adaptable DL framework (Detectron2) for diverse digital pathology tasks, reducing the need for specialized tools for each problem.
  • Transfer Learning Efficacy: Leveraging pre-trained models significantly accelerates development and improves performance, especially when dealing with limited annotated medical data.
  • Hyperparameter Tuning Importance: While a manual approach was used, the study underscores that careful hyperparameter optimization is crucial for maximizing model accuracy, even with a robust framework.
  • Data Preprocessing Nuances: The effectiveness of DL models is highly dependent on appropriate data preparation, including subimage splitting and annotation formatting, which can be dataset-specific.
  • Challenges of Imbalanced Data: Mitosis detection, in particular, presents an imbalanced data problem (rare events), necessitating strategies like data augmentation and hard negative mining, which the paper discusses as potential improvements.

Key Results

  • TUPAC16 (Object Detection): The optimized model achieved an average precision (AP) of 65.02% on the validation set and 65.14% on the testing subset, indicating successful avoidance of overfitting. The F-score was 0.628. This performance placed the model fifth among challenge entries.
  • JPATHOL (Instance Segmentation):
    • On the validation set, the model achieved APs of 19.53% for nuclei, 5.15% for epithelium, and 35.22% for tubules. The combined nuclei/epithelium/tubules task yielded 8.03% AP.
  • On the test set, the combined model achieved an AP of 14.41%. Specific APs for nuclei, epithelium, and tubules were 15.26%, 2.06%, and 25.91%, respectively.
  • Qualitative Results: Visualizations show the model's ability to detect and segment objects, though some inaccuracies and overlapping predictions were observed, particularly in the instance segmentation tasks.

Limitations and Future Work

  • Limitations:
    • Limited Data Augmentation: The initial experiments did not extensively utilize data augmentation, which could have improved model robustness and generalization.
  • Manual Hyperparameter Search: The manual approach to hyperparameter tuning can be time-consuming and may lead to local minima.
  • Staining Variability: While addressed to some extent, the inherent variability in tissue staining remains a challenge.
  • Instance Segmentation Performance: The performance on the JPATHOL instance segmentation tasks was considered mediocre, with challenges in detecting smaller instances and epithelium.
  • Limited Dataset Size: The study notes that some entries in similar challenges used external datasets to augment their training data, which was not done here.
  • Future Work:
    • Data Augmentation: Incorporating rotation, flipping, and scaling for training data.
  • Automated Hyperparameter Search: Utilizing random search or Bayesian optimization to find optimal hyperparameters more efficiently.
  • Exploring Different Architectures: Testing various pre-trained CNN models available within Detectron2.
  • Staining Normalization Techniques: Investigating alternative methods to address tissue appearance variability.
  • Improving Annotation Quality: Extracting additional mitosis instances and potentially using unsupervised methods.
  • Explainable AI: Implementing techniques to make the DL model's internal processes more transparent.
  • Multi-Model Approach: Training separate models for each JPATHOL sub-dataset and combining them for potentially better performance.
  • Larger Viewing Area: For larger entities like epithelium and tubules, exploring models that can process larger image contexts.

Summary updated

Abstract

Detection of regions of interest (ROIs) in whole slide images (WSIs) in a clinical setting is a highly subjective and a labor-intensive task. In this work, recent developments in machine learning and computer vision algorithms are presented to assess their possible usage and performance to enhance and accelerate clinical pathology procedures, such as ROI detection in WSIs. In this context, a state-of-the-art deep learning framework (Detectron2) was trained on two cases linked to the TUPAC16 dataset for object detection and on the JPATHOL dataset for instance segmentation. The predictions were evaluated against competing models and further possible improvements are discussed.
BibTeX
@article{amerikanos2022image,
  author = {Paris Panagiotis Amerikanos and Ilias Maglogiannis},
  title = {{Image Analysis in Digital Pathology Utilizing Machine Learning and Deep Neural Networks}},
  journal = {Journal of Personalized Medicine},
  year = {2022},
  doi = {10.3390/jpm12091444},
}
Figure