Rethinking Mitosis Detection: Towards Diverse Data and Feature Representation

Hao Wang, Jiatai Lin, Danyi Li, Jing Wang, Bingchao Zhao, Zhenwei Shi, Xipeng Pan, Huadeng Wang, Bingbing Li, Changhong Liang, Guoqiang Han, Li Liang, Chu Han, Zaiyi LiuarXiv 2023 · OpenReview

TL;DR

This paper introduces MitDet, a novel framework for mitosis detection in histopathology images that tackles the inherent challenges of data imbalance and feature heterogeneity. By focusing on balancing data diversity and enhancing feature representation, MitDet achieves state-of-the-art performance across multiple datasets with minimal point annotations, offering a more generalizable and efficient solution for computational pathology.

What Problem Does This Paper Address?

Mitosis detection is a critical task in computational pathology, serving as a key indicator for cancer grading and prognosis. However, it's a notoriously difficult problem due to several factors:

  • Data Imbalance: Mitotic cells are rare, making up a tiny fraction of all cells in a sample (around 0.3%). This creates a significant imbalance between positive (mitotic) and negative (non-mitotic) samples.
  • Cellular Heterogeneity: Both mitotic and non-mitotic cells exhibit considerable variation in their appearance. Mitotic cells go through distinct phases (prophase, metaphase, anaphase, telophase), and non-mitotic cells in tumors can look very diverse. This makes it hard to distinguish them, even for human pathologists.
  • Domain Shift: Differences in staining reagents, imaging protocols, and scanner types across different institutions lead to variations in image appearance (color, texture). This "domain shift" makes models trained on one dataset perform poorly on data from another.

Existing approaches often try to overcome these challenges by increasing model complexity. However, this can lead to overfitting and limited generalizability. The authors argue that a deeper understanding of the biological and data characteristics is needed.

Methods

The paper proposes a novel framework called MitDet, which rethinks mitosis detection by focusing on data and feature representation. Key components include:

  • Hematoxylin-based Nuclei Localization: Leveraging the biological fact that hematoxylin stains nuclei blue, this method uses color decomposition to extract the hematoxylin channel. This allows for label-free, high-sensitivity localization of all nuclei, providing candidate patches for further analysis. This is a significant advantage as it doesn't require manual nuclei annotations.
  • Diversity-Guided Sample Balancing (DGSB): This module addresses the data imbalance problem not just by quantity but also by diversity and difficulty.
  • 1st Sampler (Quantity and Diversity): Negative samples are clustered based on their feature vectors (extracted using a pre-trained ResNet). DGSB then samples an equal number of negative patches from each cluster, ensuring representation of diverse appearances.
  • 2nd Sampler (Difficulty): To further refine the training set, easy negative samples (those the model confidently classifies as non-mitotic) are filtered out, leaving harder negative samples that are more challenging to distinguish.
  • Stain Enhancement (SE): To combat domain shift, SE enhances diversity at both data and feature levels.
  • Unsupervised Stain Augmentation (UnSA): Using a method called RestainNet, the training data is augmented into multiple color domains without requiring manual annotations. This enriches the dataset with variations seen across different institutions.
  • EFDMix: This technique augments feature representations by mixing feature distributions from different color domains, making the model more robust to color variations.
  • Inter- and Intra- Class Diversity-Preserved Mitosis Classification (InCDP): This module tackles the heterogeneity challenge by creating a more diverse feature representation. Instead of a simple binary classification (mitotic/non-mitotic), InCDP divides each class into multiple "child" classes. This encourages the model to learn finer distinctions, pushing apart features of different classes (inter-class) and pulling together features of the same class (intra-class) using focal loss and center loss.

Datasets

The paper utilizes several publicly available datasets for training and evaluation:

  • MIDOG2021: Used for training and internal validation. This dataset is highlighted as being large-scale and having significant domain discrepancy.
  • AMIDA2013, MITOSIS2014, TUPAC2016, MIDOG2022: Used as external test sets to evaluate the model's generalizability.

The authors note that MIDOG2021 and MIDOG2022 are particularly important as they are large-scale datasets designed to test domain generalization.

Key Insights and Ideas

  • Rethinking the Problem: The core insight is that simply increasing model complexity isn't the answer to mitosis detection's challenges. Instead, a deeper understanding of the data's biological characteristics and inherent diversity is crucial for building generalizable models.
  • Biological Knowledge is Key: Leveraging the fact that mitosis is a nuclear process and hematoxylin stains nuclei allows for a powerful, label-free nuclei localization step, significantly reducing annotation burden.
  • Diversity is More Than Just Quantity: Balancing data involves not only having enough samples but also ensuring a diverse range of appearances and difficulty levels, especially for negative samples.
  • Feature Representation Matters: A simple binary classification is insufficient for the highly heterogeneous nature of mitotic cells. Creating a more granular, diverse feature space is essential for accurate differentiation.
  • Unsupervised Augmentation for Generalization: Augmenting data and features in an unsupervised manner (UnSA and EFDMix) is a flexible and effective way to improve robustness to domain shift without requiring manual labeling for each new domain.

Key Results

  • State-of-the-Art Performance: MitDet achieves state-of-the-art (SOTA) performance on the internal MIDOG2021 dataset and also on four external datasets, demonstrating superior generalizability.
  • Minimal Annotation Effort: The model achieves SOTA results using only point annotations, significantly reducing the annotation burden compared to pixel-level or bounding-box methods.
  • Significant Performance Gains: MitDet outperforms previous SOTA methods by a notable margin, particularly on datasets with larger domain discrepancies. For example, it shows a ~1% improvement in F1 score over FMDet on the internal dataset and significant boosts on external datasets like MITOSIS2014 (around 2% F1 score improvement).
  • Effectiveness of Novelties: Ablation studies confirm that each proposed module (DGSB, SE, InCDP) contributes to improved performance, with combinations yielding even better results. The final model shows substantial improvements, especially on out-of-domain (OOD) datasets.
  • High Sensitivity in Nuclei Localization: The hematoxylin-based nuclei localization achieves very high sensitivity (e.g., 0.983 on MIDOG2021 val) while filtering out over 99% of negative samples, enhancing efficiency.

Limitations and Future Work

  • Hematoxylin Localization Limitations: While highly effective, the hematoxylin-based nuclei localization can miss cells with very low staining intensity or fragmented appearances. The authors acknowledge this and suggest that future work could involve replacing this step with a more sensitive nuclei localization method if available.
  • Flexibility of Components: The authors emphasize that each component of MitDet is modular and can be replaced with superior alternatives as they become available (e.g., a better nuclei localizer, a smarter data sampler, or a more robust classifier). This suggests a clear path for future improvements.
  • Further Exploration of Diversity: While the paper introduces diversity in data and features, further research could explore more sophisticated methods for capturing and leveraging this diversity.

Summary updated

Abstract

Mitosis detection is one of the fundamental tasks in computational pathology, which is extremely challenging due to the heterogeneity of mitotic cell. Most of the current studies solve the heterogeneity in the technical aspect by increasing the model complexity. However, lacking consideration of the biological knowledge and the complex model design may lead to the overfitting problem while limited the generalizability of the detection model. In this paper, we systematically study the morphological appearances in different mitotic phases as well as the ambiguous non-mitotic cells and identify that balancing the data and feature diversity can achieve better generalizability. Based on this observation, we propose a novel generalizable framework (MitDet) for mitosis detection. The data diversity is considered by the proposed diversity-guided sample balancing (DGSB). And the feature diversity is preserved by inter- and intra- class feature diversity-preserved module (InCDP). Stain enhancement (SE) module is introduced to enhance the domain-relevant diversity of both data and features simultaneously. Extensive experiments have demonstrated that our proposed model outperforms all the SOTA approaches in several popular mitosis detection datasets in both internal and external test sets using minimal annotation efforts with point annotations only. Comprehensive ablation studies have also proven the effectiveness of the rethinking of data and feature diversity balancing. By analyzing the results quantitatively and qualitatively, we believe that our proposed model not only achieves SOTA performance but also might inspire the future studies in new perspectives. Source code is at https://github.com/Onehour0108/MitDet.
BibTeX
@misc{wang2023rethinking,
  author = {Hao Wang and Jiatai Lin and Danyi Li and Jing Wang and Bingchao Zhao and Zhenwei Shi and Xipeng Pan and Huadeng Wang and Bingbing Li and Changhong Liang and Guoqiang Han and Li Liang and Chu Han and Zaiyi Liu},
  title = {{Rethinking Mitosis Detection: Towards Diverse Data and Feature Representation}},
  year = {2023},
  eprint = {2307.05889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.05889},
}
Figure