TL;DR
Researchers have created a comprehensive, large-scale dataset of annotated whole slide images (WSIs) of canine cutaneous mast cell tumors, specifically focusing on mitotic figures. This dataset, the largest of its kind, aims to improve the accuracy and reliability of automated mitosis detection in cancer diagnostics, a crucial but challenging task.
What Problem Does This Paper Address?
Accurately identifying and counting mitotic figures (cells undergoing division) in tumor tissue is vital for cancer prognosis and treatment planning. However, manual counting is notoriously difficult, time-consuming, and prone to significant inter- and intra-rater variability. This is due to the subtle visual cues of mitotic cells, their variable appearance across different cell cycle phases, and their uneven distribution within a tumor. While automated methods show promise, they have historically lacked large, comprehensively annotated datasets of whole slide images to train and validate against. This paper addresses this gap by providing such a dataset, enabling more robust development and evaluation of algorithms for mitotic figure detection.
Methods
The authors developed a multi-stage annotation process to create their dataset. Initially, two veterinary pathology experts manually annotated mitotic figures and other cell types (neoplastic mast cells, inflammatory granulocytes, and mitotic figure look-alikes) on 32 whole slide images. To address the risk of missed annotations, they then employed an algorithm-aided approach. This involved training a deep learning model (ResNet-18) to classify cells and a RetinaNet object detector to identify potential mitotic figure candidates. These algorithm-generated suggestions were then reviewed by human experts, leading to the creation of two augmented dataset variants: the Hard-Example Augmented Expert Labelled (HEAEL) dataset and the Object-Detection Augmented Expert Labelled (ODAEL) dataset. The ODAEL variant, in particular, aimed to capture potentially missed mitotic figures and "hard negative" examples (cells that resemble mitotic figures but are not).
Datasets
The core dataset consists of 32 whole slide images (WSIs) of canine cutaneous mast cell tumors. These WSIs are provided in the Aperio SVS format. The paper introduces three distinct annotation variants derived from these WSIs:
- Manually Expert Labelled (MEL) dataset: The baseline dataset with manual annotations by two experts.
- Hard-Example Augmented Expert Labelled (HEAEL) dataset: An augmentation of the MEL dataset where ambiguous cells were further classified using a ResNet-18 based classifier.
- Object-Detection Augmented Expert Labelled (ODAEL) dataset: The most comprehensive variant, generated using a RetinaNet object detector to identify potential mitotic figures, which were then reviewed by experts. This dataset includes the largest number of annotations.
These datasets are made available as SQLite3 database files. The paper also references and compares its dataset to existing benchmarks like MITOS 2012, MICCAI AMIDA 13, MITOS-ATYPIA 2014, and TUPAC 2016.
Key Insights and Ideas
- The critical need for large-scale, WSI-level annotations: The paper strongly emphasizes that previous datasets, often limited to small regions or specific fields of view, are insufficient for truly robust mitotic figure detection algorithms that can operate on entire pathology slides.
- Algorithm-assisted annotation significantly improves coverage: By leveraging deep learning models to propose candidates, the authors were able to identify many more mitotic figures and "hard negative" examples than manual annotation alone, leading to a more comprehensive dataset.
- Mitotic figure look-alikes are a significant challenge: The dataset includes a substantial number of "mitotic figure look-alikes," highlighting the difficulty even for expert annotators and automated systems to distinguish true mitotic figures from other cellular structures. This category is particularly enriched in the augmented datasets.
- The value of "hard examples" for model training: The HEAEL dataset specifically focuses on identifying and classifying "hard examples," demonstrating a strategy to improve model performance by explicitly training on challenging cases.
Key Results
- The dataset contains a total of 262,481 annotations across all variants, with 44,880 specifically identified as mitotic figures.
- For algorithmic validation, the authors achieved F1-scores of 0.786 for the manually labelled dataset and 0.820 for the algorithm-aided dataset (ODAEL).
- A ResNet-18 based cell classifier achieved an accuracy of 91.390% on classifying preselected cell patches, with the main confusion occurring between mitotic figures and mitotic figure look-alikes.
- The ODAEL variant consistently yielded the highest F1 scores in detection tasks, underscoring the benefit of the algorithm-augmented annotation process.
Limitations and Future Work
- Risk of missing candidates: Despite the algorithm-aided approach, the authors acknowledge that manual screening of large images inherently carries a risk of missing candidates, which they perceive as a main risk to data quality.
- Potential bias from sampling scheme: The authors note that their customized sampling scheme for training might lead to an overestimation of mitotic figure likelihood by the model, necessitating careful threshold optimization.
- Future work: The authors suggest that this dataset will enable the development and refinement of data-driven algorithms for mitotic figure identification on complete WSIs, as well as region-of-interest detection algorithms. They also highlight the potential for using this dataset to investigate the impact of dataset size and annotation quality on model performance.
Summary updated