TL;DR
This paper introduces MIDOG++, a comprehensive dataset for mitotic figure detection in histopathology images, featuring diverse tumor types, labs, and scanners to address the critical challenge of domain shift in deep learning models. They demonstrate that while single-domain training struggles, models trained on this diverse dataset show improved generalizability, highlighting the importance of broad data representation for robust AI in pathology.
What Problem Does This Paper Address?
Accurately counting mitotic figures (cells undergoing division) in tumor tissue is crucial for predicting how aggressive a cancer is and guiding treatment. However, manually counting these figures is time-consuming and prone to significant variability between pathologists. While deep learning offers a promising automated solution, these models often falter when applied to data that differs from their training set – a phenomenon known as "domain shift." This shift can arise from various factors, including different tumor types, variations in how tissue is prepared and stained, and the use of different whole-slide imaging scanners. Existing datasets often lack this crucial diversity, limiting the real-world applicability of automated mitotic figure detection. This paper tackles this problem by creating a dataset that explicitly captures this multi-domain variability.
Methods
The researchers developed the MIDOG++ dataset by extending previous MIDOG challenge datasets. They collected 503 histological specimens from seven different tumor types (human breast carcinoma, lung carcinoma, lymphosarcoma, neuroendocrine tumor, cutaneous mast cell tumor, cutaneous melanoma, and soft tissue sarcoma). These specimens were processed across multiple laboratories and digitized using five different whole-slide scanners, introducing significant domain variability.
For annotation, a pathologist meticulously screened each image twice, marking mitotic figures and "hard negatives" (structures that resemble mitotic figures but are not). To ensure accuracy and reduce missed detections, a deep learning model was used to identify potential candidates missed during manual screening. Finally, a three-expert consensus approach was employed, where a second and third pathologist reviewed disagreements to establish a high-quality ground truth.
To evaluate the dataset's utility, they trained a RetinaNet object detection model. They conducted experiments using:
- Single-domain training: Training on one tumor type and testing on all others to assess cross-domain performance.
- Leave-one-domain-out training: Training on all but one tumor type to evaluate generalizability when a specific domain is absent from training.
- Full training: Training on the entire dataset.
Datasets
The core contribution is the MIDOG++ dataset, which includes:
- 503 histological specimens.
- 7 different tumor types (breast carcinoma, lung carcinoma, lymphosarcoma, neuroendocrine tumor, cutaneous mast cell tumor, cutaneous melanoma, and soft tissue sarcoma).
- Specimens processed in multiple laboratories.
- Digitized using five different whole-slide scanners.
- A total of 11,937 annotated mitotic figures.
This dataset is an extension of the MIDOG 2021 and 2022 challenge datasets, specifically designed to incorporate a wide range of domain shifts.
Key Insights and Ideas
- Domain shift is a major hurdle: The study strongly reinforces that deep learning models trained on limited domains perform poorly when encountering new data variations (different tumor types, scanners, labs).
- Diversity is key for generalization: Training on a dataset that encompasses a wide range of variations (like MIDOG++) significantly improves a model's ability to generalize to unseen data.
- Mitotic figure detection is inherently challenging: Even with expert annotation, there's a notable inter-rater disagreement, highlighting the complexity of the task.
- Species differences might be less impactful than other domain shifts: The study suggests that models can generalize across species (human, canine, feline) reasonably well, provided other domain variations are also addressed.
- Neuroendocrine tumors present a unique challenge: This tumor type showed consistently low performance, likely due to class imbalance (fewer mitotic figures) and inherent morphological complexities.
Key Results
- Single-domain training shows significant performance drops: When trained on one tumor type and tested on others, F1 scores dropped considerably, with some cross-domain performances as low as 0.09 (canine lymphosarcoma model tested on human astrocytoma).
- Leave-one-domain-out training improves generalizability: Models trained on a broader set of domains (all but one) showed better performance across various test domains compared to single-domain training.
- Human melanoma and canine mast cell tumor domains showed high F1 scores (0.81) in leave-one-out settings: This suggests these domains might be more representative or easier to generalize from.
- Canine lymphosarcoma exhibited the strongest domain shift: Models trained on this domain generalized poorly to others, and models tested on it often performed worst, likely due to its distinct morphology and smaller cell size.
- Evaluation on the MIDOG 2022 test set confirmed findings: The models demonstrated limited generalization on unseen data, but training on the full MIDOG++ dataset showed better performance than single-domain training.
Limitations and Future Work
- Class Imbalance: The human neuroendocrine tumor domain suffers from a low count of mitotic figures, which negatively impacts model training and performance. The authors suggest this makes the object detection problem significantly more challenging.
- Difficulty in Separating Domain Shift Sources: While the dataset captures variations from species, tumor type, lab, and scanner, it's challenging to isolate the exact impact of each individual factor in experiments.
- Future Work: The authors implicitly suggest that further development of robust domain generalization techniques is needed. The creation of such a diverse dataset is a foundational step for future research into more resilient AI models for pathology. They also provide code and data to facilitate further research.
Summary updated