Superpixel Pre-Segmentation of HER2 Slides for Efficient Annotation

Mathias Öttl, Jana Mönius, Christian Marzahl, Matthias Rübner, Carol I. Geppert, Arndt Hartmann, Matthias W. Beckmann, Peter Fasching, Andreas Maier, Ramona Erber, Katharina BreiningerarXiv 2022 · arXiv

TL;DR

This paper proposes a method to accelerate the tedious process of annotating HER2-stained breast cancer slides by using superpixel pre-segmentation. By combining domain-specific color adjustments or deep learning embeddings with hierarchical clustering, the researchers achieved a 73% improvement in boundary accuracy over standard methods, providing a high-quality "first draft" for pathologists to refine.

What Problem Does This Paper Address?

In breast cancer diagnostics, determining the HER2 (Human Epidermal growth factor Receptor 2) status is critical for choosing the right therapy. While deep learning can help automate this analysis, training these models requires massive amounts of fine-grained, manually annotated data.

For a pathologist, tracing complex cell membranes and tumor boundaries is incredibly time-consuming. Standard "superpixel" algorithms—which group pixels into meaningful atomic regions—could help by providing a pre-segmented map that an expert simply needs to correct. However, standard algorithms like SLIC (Simple Linear Iterative Clustering) often struggle with the non-uniform textures and varying scales of tissue structures in histopathology, leading to boundaries that don't align with actual biological structures.

Methods

The authors compared four different ways to generate superpixels and introduced a novel merging step to handle varying region sizes:

  1. Standard SLIC (Baseline): The industry-standard algorithm using RGB color and spatial proximity.
  2. Adjusted SLIC: A domain-aware version that converts images to the HED (Haematoxylin-Eosin-DAB) color space. It specifically filters out the hematoxylin channel (which highlights cell nuclei) to force the algorithm to focus on the HER2-stained membranes.
  3. Pre-trained ResNet-50: Uses feature embeddings from a ResNet-50 (trained on the COCO dataset) as the input for the clustering.
  4. HER2 Autoencoder: A custom denoising autoencoder trained specifically on HER2 slides to learn a robust, 64-dimensional internal representation of the tissue.

Hierarchical Clustering: To solve the problem of "oversegmentation" (where one biological structure is broken into too many tiny pieces), the authors applied agglomerative clustering. This merges neighboring superpixels that are similar in their feature space, allowing the segmentation to adapt to both large homogeneous areas and small, detailed structures.

Datasets

  • Source: 20 tissue sections from 20 different patients, covering the full range of HER2 scores (0, 1+, 2+, and 3+).
  • Images: 84 high-resolution image patches (1.5 mm × 1.5 mm) extracted from Whole Slide Images (WSI) scanned at 20x.
  • Ground Truth: Four test patches were meticulously hand-annotated by a medical student and reviewed by a board-certified pathologist to serve as the gold standard for evaluation.

Key Insights and Ideas

  • Domain Knowledge Trumps Generic Features: The Adjusted SLIC (using HED color space) significantly outperformed the baseline. Interestingly, the generic ResNet-50 features performed poorly, suggesting that features learned from everyday objects (like those in the COCO dataset) don't translate well to the microscopic world of pathology.
  • The Power of Merging: Hierarchical clustering was the most impactful addition. By allowing superpixels to merge based on content, the system effectively "discovered" larger biological regions, drastically improving how well the boundaries matched the ground truth.
  • Targeted Embeddings: The denoising autoencoder, despite being a relatively simple architecture, provided the best feature space for segmentation because it was trained specifically on the textures of HER2-stained slides.

Key Results

The researchers evaluated their methods using the Boundary F1 score, which measures how closely the superpixel edges align with the expert's annotations.

  • Stand-alone Improvement: The autoencoder-based superpixels achieved a 23% increase in boundary F1 score over the baseline SLIC.
  • The "Aha" Moment: When hierarchical clustering was added to the Adjusted SLIC and the Autoencoder methods, the boundary F1 score jumped by 73% compared to the baseline.
  • Visual Accuracy: As seen in the plots below, the clustering methods (solid lines) maintain much higher accuracy even as the superpixel size increases.
Fig. 1. Plots of achievable segmentation accuracy (ASA) (a) and boundary F1 score (b) for the different methods for different superpixel sizes.

Visually, the difference is striking. In Figure 2, the blue lines (clustered) follow the biological structures much more faithfully than the black lines (initial superpixels).

Fig. 2. Results for the superpixel methods at a inital superpixel diameter of 120 pixels. The initial superpixels are drawn with black outlining, while the clustered superpixels are visualized in blue.

Limitations and Future Work

While the results are a strong proof-of-concept, the authors note a few hurdles:

  • Dataset Size: The evaluation was performed on a small number of fully annotated patches (four), which limits the statistical power of the results.
  • Hardware Bias: All images came from a single scanning device; future work needs to test if these models hold up across different labs and scanners.
  • The "Human in the Loop" Test: The ultimate goal is to save time. While the mathematical accuracy is higher, the authors have yet to conduct a user study to quantify exactly how many minutes or hours this saves a pathologist in a real-world workflow.

Summary updated

Abstract

Supervised deep learning has shown state-of-the-art performance for medical image segmentation across different applications, including histopathology and cancer research; however, the manual annotation of such data is extremely laborious. In this work, we explore the use of superpixel approaches to compute a pre-segmentation of HER2 stained images for breast cancer diagnosis that facilitates faster manual annotation and correction in a second step. Four methods are compared: Standard Simple Linear Iterative Clustering (SLIC) as a baseline, a domain adapted SLIC, and superpixels based on feature embeddings of a pretrained ResNet-50 and a denoising autoencoder. To tackle oversegmentation, we propose to hierarchically merge superpixels, based on their content in the respective feature space. When evaluating the approaches on fully manually annotated images, we observe that the autoencoder-based superpixels achieve a 23% increase in boundary F1 score compared to the baseline SLIC superpixels. Furthermore, the boundary F1 score increases by 73% when hierarchical clustering is applied on the adapted SLIC and the autoencoder-based superpixels. These evaluations show encouraging first results for a pre-segmentation for efficient manual refinement without the need for an initial set of annotated training data.
BibTeX
@misc{ottl2022superpixel,
  author = {Mathias Öttl and Jana Mönius and Christian Marzahl and Matthias Rübner and Carol I. Geppert and Arndt Hartmann and Matthias W. Beckmann and Peter Fasching and Andreas Maier and Ramona Erber and Katharina Breininger},
  title = {{Superpixel Pre-Segmentation of HER2 Slides for Efficient Annotation}},
  year = {2022},
  eprint = {2201.07572},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.07572},
}
Figure