A Guided Spatial Transformer Network for Histology Cell Differentiation

Marc Aubreville, Maximilian Krappmann, Christof Bertram, Robert Klopfleisch, Andreas K. Maier

TL;DR

This paper introduces a Guided Spatial Transformer Network (STN) designed to automatically localize and classify cells in histology slides, specifically targeting mitotic figures. By training on a newly developed dataset of 10,000 mitotic figures—ten times larger than previous benchmarks—the authors achieved a 91.45% classification accuracy, providing a more objective tool for tumor grading.

What Problem Does This Paper Address?

In cancer diagnostics, pathologists count "mitotic figures" (cells undergoing division) to determine tumor aggressiveness. However, this task is a nightmare for human consistency:

  1. Tedium: Slides contain thousands of cells; manually inspecting them all is impossible.
  2. Subjectivity: Pathologists often disagree on what constitutes a mitotic figure versus a normal cell or a granulocyte.
  3. Data Scarcity: Previous deep learning models struggled because public datasets were tiny (often <1,000 examples), making it hard for models to generalize across the high visual variance of different mitotic phases (prophase, metaphase, etc.).

The authors aim to bridge this gap by creating a massive dataset and a specialized neural network architecture that handles both the "where" (localization) and the "what" (classification) of cell detection.

Methods

The core innovation is the Guided Spatial Transformer Network. Standard CNNs are often "spatially invariant" to a fault or require perfectly centered input. The authors use an STN to solve this:

  • The Localizer: A sub-network that looks at a large, uncentered image patch and regresses the parameters of an affine transformation (translation and scaling). It essentially "finds" the cell and crops it.
  • The "Guided" Aspect: Unlike typical STNs that learn to zoom/rotate purely to improve classification, this model is guided during training. The authors used known offsets from their preprocessing stage to calculate a specific localization loss ($l_{loc}$). This forces the localizer to actually center the cell rather than finding some arbitrary local optimum.
  • The Classifier: A 7-layer CNN (including Inception-style blocks) that receives the "corrected" image from the STN and classifies it into one of three categories: mitotic tumor cells, non-mitotic tumor cells, or granulocytes.
  • Three-Stage Training: They first train the classifier on centered images, then the localizer on offset images, and finally fine-tune the entire pipeline with a combined loss function.

Datasets

The authors introduced a significant new dataset derived from 12 canine mast cell tumors:

  • Scale: ~37,800 single cell annotations.
  • Composition: 10,800 mitotic figures, 10,400 granulocytes, and 16,600 normal tumor cells.
  • Significance: This is roughly 10x larger than previous standard datasets like MITOS-ATYPIA-14 or AMIDA13. By purposely selecting a balanced number of cells from each category (rather than following natural prevalence), they ensured the model had enough examples of rare mitotic events to learn effectively.

Key Insights and Ideas

  • Guidance Prevents "Drift": On complex histology slides, an unguided STN might get stuck on "local extrema"—focusing on a random dark spot that helps classification but isn't the actual cell. Explicitly guiding the STN with ground-truth coordinates ensures the model learns meaningful spatial features.
  • Inception Blocks for Scale: The inclusion of Inception layers (which process information at multiple scales simultaneously) was found to significantly improve convergence for both the localizer and the classifier.
  • Prioritizing Sparse Events: By balancing the training set, the authors inherently prioritize the detection of rare, high-importance events (mitosis) over common ones (normal cells), which is exactly what a pathologist needs.

Key Results

The model was evaluated using five-fold cross-validation and compared against a baseline 12-layer CNN (similar to the state-of-the-art Cireșan architecture of that era).

  • Accuracy: The Guided STN achieved 91.45% mean accuracy, a significant jump over the baseline's 86.9%.
  • Precision/Recall: Precision ranged from 90.4% to 93.4%, and recall from 90.1% to 92.8% across the three cell types.
  • Robustness: The model proved highly capable of distinguishing between mitotic figures and granulocytes, which are notoriously easy to confuse in histology.

Limitations and Future Work

  • Selection Bias: Because the authors purposely selected candidates rather than annotating every single cell on a whole slide, the dataset might not perfectly represent the "real world" distribution of cell appearances.
  • Localizer "Errors": In some cases, the localizer focused on a different cell near the annotated one. While the authors argue this isn't a failure (if the nearby cell is also a tumor cell), it highlights the difficulty of single-cell localization in dense tissue.
  • Future Directions: The authors suggest using this large dataset for transfer learning to improve performance on smaller, fully-labeled competition datasets. They also envision embedding this algorithm directly into digital microscopes to assist pathologists in real-time.

Summary updated

Abstract

Identification and counting of cells and mitotic figures is a standard task in diagnostic histopathology. Due to the large overall cell count on histological slides and the potential sparse prevalence of some relevant cell types or mitotic figures, retrieving annotation data for sufficient statistics is a tedious task and prone to a significant error in assessment. Automatic classification and segmentation is a classic task in digital pathology, yet it is not solved to a sufficient degree. We present a novel approach for cell and mitotic figure classification, based on a deep convolutional network with an incorporated Spatial Transformer Network. The network was trained on a novel data set with ten thousand mitotic figures, about ten times more than previous data sets. The algorithm is able to derive the cell class (mitotic tumor cells, non-mitotic tumor cells and granulocytes) and their position within an image. The mean accuracy of the algorithm in a five-fold cross-validation is 91.45 %. In our view, the approach is a promising step into the direction of a more objective and accurate, semi-automatized mitosis counting supporting the pathologist.
BibTeX
@inproceedings{aubreville2017guided,
  author = {Marc Aubreville and Maximilian Krappmann and Christof Bertram and Robert Klopfleisch and Andreas K. Maier},
  title = {{A Guided Spatial Transformer Network for Histology Cell Differentiation}},
  booktitle = {VCBM},
  year = {2017},
  url = {https://openreview.net/forum?id=YfI2N3pj_mQ},
}
Figure