Effects of annotation granularity in deep learning models for histopathological images

Jiangbo Shi, Zeyu Gao, Haichuan Zhang, Pargorn Puttapirat, Chunbao Wang, Xiangrong Zhang, Chen Li

Abstract

Pathological examination is crucial to cancer diagnosis. Typically, pathologists draw conclusions based on observed cell and tissue structures on histology slides. Rapid developments in machine learning, especially deep learning, have established robust and accurate classifiers to assist in this process. Most machine learning systems rely heavily on annotated datasets to perform tasks such as classification and segmentation. This work investigates different granularities of annotations in histopathological datasets—including image-wise, bounding box, ellipse-wise, and pixel-wise—to verify their influence on deep learning models. Our experiments show that state-of-the-art classifiers perform significantly better when trained on pixel-wise annotation datasets, with average improvements in precision, recall, and F1-score of 7.87%, 8.83%, and 7.85%, respectively. Similarly, semantic segmentation algorithms achieve 8.33% better accuracy with pixel-wise annotations. These findings suggest that finer-grained annotations not only improve model performance but also help extract more accurate phenotypic information, potentially aiding pathologists in more precise diagnosis.

Introduction

Pathological examination is the "gold standard" for cancer diagnosis, but it is a grueling, manual process. As cancer rates rise globally, the workforce of pathologists is shrinking, leading to massive increases in diagnostic workloads. While deep learning (DL) offers a solution through automated image analysis, these models are only as good as the data used to train them.

The core challenge lies in annotation granularity. Labeling every single pixel of a tumor cell (pixel-wise) is incredibly time-consuming for a trained pathologist, whereas drawing a rough square around it (bounding box) is much faster. This study investigates whether the extra effort of fine-grained labeling results in a superior model or if simpler, coarser annotations are sufficient for clinical applications.

Methodology

Granularity Levels and Models

The researchers conducted a comparative study across two primary tasks: Classification (identifying cancer grade) and Semantic Segmentation (outlining cell boundaries). They compared three types of labels:

  1. Bounding Box (Bb): A simple rectangle around the cell.
  2. Ellipse-wise (el): An oval fitting the cell shape.
  3. Pixel-wise (pw): A precise mask covering every pixel of the cell nucleus.
Comparison of annotation granularity: (a) Bounding box, (b) Ellipse-wise, and (c) Pixel-wise.

To ensure a fair comparison, the team used a "downgrade" approach: starting with high-quality pixel-wise labels and using OpenCV to mathematically generate ellipses and bounding boxes, ensuring the underlying image data remained identical. They tested standard architectures including ResNet18, VGG16, and MobileNet for classification, and UNet, FPN, and LinkNet for segmentation.

Datasets

The study primarily used images of clear cell renal carcinoma (ccRCC) sourced from The Cancer Genome Atlas (TCGA). Six rigorously trained experts annotated 200 diagnostic regions (800x800 pixels), grading cells according to the WHO/ISUP standard (Grades 1 and 2).

Results and Discussion

Quantitative Performance Gains

The shift from bounding boxes to pixel-wise annotations yielded significant quantitative gains across all tested models:

  • Classification: On average, pixel-wise labels improved precision by 7.87%, recall by 8.83%, and F1-score by 7.85%.
  • Segmentation Accuracy: In the UNet model, pixel-wise annotations achieved 95.79% accuracy—6.2% higher than bounding box labels.
Semantic segmentation results on histopathological slides using granular annotations to identify cell phenotypes.

Precision and Interpretability

Finer annotations allow models to extract more accurate "phenotypic information"—the specific physical characteristics like color, texture, and morphology of a cell. Coarse boxes often include "noise" from background tissue that can confuse the model. Furthermore, pixel-wise training enhances interpretability; by providing precise outlines, the model can show a pathologist exactly which features led to a diagnosis, thereby increasing clinical trust.

Conclusion

While pixel-wise labels are superior, they are undeniably labor-intensive to produce. However, the performance gains and the ability to extract accurate phenotypic information suggest that finer-grained annotation is a worthwhile investment for developing reliable clinical tools. Future work will explore genotype-phenotype associations, linking these precise visual features to underlying genetic mutations, and the potential for automated pathological reporting.

Summary updated

Abstract

Pathological is crucial to cancer diagnosis. Usually, Pathologists draw their conclusion based on observed cell and tissue structure on histology slides. Rapid development in machine learning, especially deep learning have established robust and accurate classifiers. They are being used to analyze histopathological slides and assist pathologists in diagnosis. Most machine learning systems rely heavily on annotated data sets to gain experiences and knowledge to correctly and accurately perform various tasks such as classification and segmentation. This work investigates different granularity of annotations in histopathological data set including image-wise, bounding box, ellipse-wise, and pixel-wise to verify the influence of annotation in pathological slide on deep learning models. We design corresponding experiments to test classification and segmentation performance of deep learning models based on annotations with different annotation granularity. In classification, state-of-the-art deep learning-based classifiers perform better when trained by pixel-wise annotation dataset. On average, precision, recall and F1-score improves by 7.87%, 8.83% and 7.85% respectively. Thus, it is suggested that finer granularity annotations are better utilized by deep learning algorithms in classification tasks. Similarly, semantic segmentation algorithms can achieve 8.33% better segmentation accuracy when trained by pixel-wise annotations. Our study shows not only that finer-grained annotation can improve the performance of deep learning models, but also help extracts more accurate phenotypic information from histopathological slides. Intelligence systems trained on granular annotations may help pathologists inspecting certain regions for better diagnosis. The compartmentalized prediction approach similar to this work may contribute to phenotype and genotype association studies.
BibTeX
@inproceedings{shi2019effects,
  author = {Jiangbo Shi and Zeyu Gao and Haichuan Zhang and Pargorn Puttapirat and Chunbao Wang and Xiangrong Zhang and Chen Li},
  title = {{Effects of annotation granularity in deep learning models for histopathological images}},
  booktitle = {BIBM},
  year = {2019},
  url = {https://openreview.net/forum?id=2N3NQUJKeGr},
}
Figure