Effects of annotation granularity in deep learning models for histopathological images

Jiangbo Shi, Zeyu Gao, Haichuan Zhang, Pargorn Puttapirat, Chunbao Wang, Xiangrong Zhang, Chen Li

TL;DR

In the world of AI-driven cancer diagnosis, the "fineness" of your data labels matters just as much as the quantity. This paper demonstrates that training deep learning models on pixel-precise annotations—rather than rough bounding boxes—significantly boosts accuracy in both classifying and segmenting cancerous cells, offering a clear path toward more reliable and interpretable clinical tools.

What Problem Does This Paper Address?

Pathological examination is the "gold standard" for cancer diagnosis, but it is a grueling, manual process. As cancer rates rise, the global workforce of pathologists is shrinking, leading to massive increases in diagnostic workloads. While deep learning (DL) offers a solution through automated image analysis, these models are only as good as the data used to train them.

The core challenge lies in annotation granularity. Labeling every single pixel of a tumor cell (pixel-wise) is incredibly time-consuming for a trained pathologist, whereas drawing a rough square around it (bounding box) is much faster. The authors set out to answer a critical question: Does the extra effort of fine-grained labeling actually result in a better model, or can AI "get the gist" from simpler, coarser annotations?

Methods

The researchers conducted a comparative study across two primary tasks: Classification (identifying the grade of cancer) and Semantic Segmentation (outlining the exact boundaries of cancer cells).

  • Granularity Levels: They compared three types of labels:
  • Bounding Box (Bb): A simple rectangle around the cell.
  • Ellipse-wise (el): An oval fitting the cell shape.
  • Pixel-wise (pw): A precise mask covering every pixel of the cell nucleus.
  • Classification Models: They tested three standard architectures: ResNet18, VGG16, and MobileNet.
  • Segmentation Models: They utilized fully convolutional networks: UNet, Feature Pyramid Network (FPN), and LinkNet.
  • The "Downgrade" Approach: To ensure a fair comparison, they started with high-quality pixel-wise labels and used OpenCV to mathematically "downgrade" them into ellipses and bounding boxes, ensuring the underlying data remained identical.

Datasets

The study primarily used images of clear cell renal carcinoma (ccRCC) sourced from the The Cancer Genome Atlas (TCGA).

  • Scale: 200 diagnostic regions (800x800 pixels) were annotated by six rigorously trained experts.
  • Grading: Cells were labeled according to the WHO/ISUP grading standard (Grades 1 and 2).
  • External Benchmarks: The paper also references existing datasets like BACH (breast cancer), CAMELYON 16 (lymph nodes), and DigestPath 2019 to provide context on how different competitions currently handle annotation granularity.

Key Insights and Ideas

  • Precision Breeds Performance: Finer annotations allow models to extract more accurate "phenotypic information"—the specific physical characteristics like color, texture, and morphology of a cell. Coarse boxes often include "noise" (background tissue) that confuses the model.
  • The Interpretability Bonus: Beyond raw accuracy, pixel-wise training makes models more interpretable. When a model is trained on precise outlines, it can show a pathologist exactly which cell features led to a diagnosis, increasing clinical trust.
  • The Patch Size "Sweet Spot": The researchers found that a 64x64 patch size often performed better than 32x32 or 128x128. Smaller patches lack context, while larger patches contain too much "mesenchyme" (connective tissue), making it harder for the model to focus on the relevant features of the nucleus.

Key Results

The shift from bounding boxes to pixel-wise annotations yielded significant quantitative gains:

  • Classification: On average, using pixel-wise labels improved Precision by 7.87%, Recall by 8.83%, and F1-score by 7.85% across all tested models.
  • Segmentation Accuracy: In the UNet model, pixel-wise annotations achieved 95.79% accuracy, which was 3.3% higher than ellipse-wise and 6.2% higher than bounding box labels.
  • Consistency: Across the board, regardless of the specific architecture (VGG, ResNet, or MobileNet), the "finer is better" rule held true, suggesting this is a fundamental property of how CNNs learn from histopathological data.

Limitations and Future Work

The authors acknowledge the "elephant in the room": cost. While pixel-wise labels are superior, they are "time-consuming and laborious" to produce at scale. The study doesn't provide a cost-benefit analysis of whether the ~8% performance gain justifies the massive increase in human labor required for pixel-perfect labeling.

Future Directions: * Genotype-Phenotype Association: The authors suggest that these precise compartmentalized predictions could eventually help link specific visual cell features (phenotypes) to underlying genetic mutations (genotypes). * Automated Reporting: They envision these models eventually helping to automatically generate pathological diagnosis reports by combining image features with natural language processing.

Summary updated

Abstract

Pathological is crucial to cancer diagnosis. Usually, Pathologists draw their conclusion based on observed cell and tissue structure on histology slides. Rapid development in machine learning, especially deep learning have established robust and accurate classifiers. They are being used to analyze histopathological slides and assist pathologists in diagnosis. Most machine learning systems rely heavily on annotated data sets to gain experiences and knowledge to correctly and accurately perform various tasks such as classification and segmentation. Generally, annotations made in pathology-related datasets have inherited annotation methods from natural scene images. This work investigates different granularity of annotations in histopathological data set including image-wise, bounding box, ellipse-wise, and pixel-wise to verify the influence of annotation in pathological slide on deep learning models. We design corresponding experiments to test classification and segmentation performance of deep learning models based on annotations with different annotation granularity. In classification, state-of-the-art deep learning-based classifiers perform better when trained by pixel-wise annotation dataset. On average, precision, recall and F1-score improves by 7.87%, 8.83% and 7.85% respectively. Thus, it is suggested that finer granularity annotations are better utilized by deep learning algorithms in classification tasks. Similarly, semantic segmentation algorithms can achieve 8.33% better segmentation accuracy when trained by pixel-wise annotations. Our study shows not only that finer-grained annotation can improve the performance of deep learning models, but also help they extract more accurate phenotypic information from histopathological slides. The accurate and spatially precise acquisitions of phenotypic information can improve the reliability of the model prediction. Intelligence systems trained on granular annotations may help pathologists inspecting certain regions and features in the slide that were mainly used to calculate the prediction. The compartmentalized prediction approach similar to this work may contribute to phenotype and genotype association studies.
BibTeX
@inproceedings{shi2019effects,
  author = {Jiangbo Shi and Zeyu Gao and Haichuan Zhang and Pargorn Puttapirat and Chunbao Wang and Xiangrong Zhang and Chen Li},
  title = {{Effects of annotation granularity in deep learning models for histopathological images}},
  booktitle = {BIBM},
  year = {2019},
  url = {https://openreview.net/forum?id=2N3NQUJKeGr},
}

Loading related content...

Figure