Stain normalization gives greater generalizability than stain jittering in neural network training for the classification of coeliac disease in duodenal biopsy whole slide images

B.a. Schreiber, Jim Denholm, Julian Gilbey, Carola‐Bibiane Schönlieb, Elizabeth J. SoilleuxJournal of Pathology Informatics 2023 · DOI

TL;DR

This paper investigates how different stain processing techniques—stain normalization and stain jittering—affect the generalizability of deep learning models for classifying coeliac disease in duodenal biopsy images. They found that stain normalization significantly improves model performance across different scanners and laboratories, while stain jittering is better for data from the same source.

What Problem Does This Paper Address?

Diagnosing coeliac disease (CD) typically relies on subjective analysis of duodenal biopsies by pathologists, which suffers from poor inter-observer agreement. While neural networks show promise for automating this diagnosis using whole slide images (WSIs), a major challenge is ensuring these models generalize well to images acquired from different scanners and laboratories. Previous work often trained and validated models on data from a single source, raising concerns about real-world applicability. This study aims to address this by comparing two stain processing techniques, stain normalization and stain jittering, to see which better enables models to generalize across diverse WSI acquisition conditions.

Methods

The researchers trained a ResNet-34 deep neural network for tile-level classification of CD. They compared three experimental setups:

  1. No Stain Processing (NP): The model was trained on raw WSI tiles without any stain modification.
  2. Stain Normalization (SN): Tiles were processed to match a fixed target stain absorption vector, aiming to standardize the appearance of the tissue.
  3. Stain Jittering (SJ): Stain absorption vectors were randomly perturbed by a small variable, encouraging the model to learn stain-independent features.

All models were trained on a dataset from a Ventana scanner and then validated on multiple datasets:

  • Ventana validation set (same scanner, same lab).
  • Hamamatsu and Aperio 1 datasets (same lab, different scanners).
  • Aperio 2 dataset (different lab, different scanner).

Performance was evaluated using WSI-level accuracy, sensitivity, specificity, precision, and F1 scores, with a focus on the F1 score and AUC (Area Under the ROC Curve) for a comprehensive assessment.

Datasets

The study utilized the following datasets:

  • Ventana Training Dataset: 331 WSIs (190 CD-positive, 141 normal) from a Ventana scanner, used for training.
  • Ventana Validation Dataset: 83 WSIs (48 CD-positive, 35 normal) from the same source as the training data.
  • Hamamatsu Dataset: 83 WSIs (48 CD-positive, 35 normal), which were rescans of the Ventana validation slides on a Hamamatsu scanner.
  • Aperio 1 Dataset: 83 WSIs (48 CD-positive, 35 normal), rescans of the Ventana validation slides on an Aperio scanner.
  • Aperio 2 Dataset: 75 WSIs (38 CD-positive, 37 normal) from a different laboratory and scanned on an Aperio scanner.

Key Insights and Ideas

  • Stain Variation is a Major Hurdle: Even when using the same scanner and lab, raw WSI data exhibits significant stain variations that can cause models to overfit and perform poorly on unseen data.
  • Stain Normalization for Generalizability: Standardizing the stain appearance (stain normalization) is crucial for building models that can reliably classify CD across different scanners and, importantly, different laboratories.
  • Stain Jittering for Intra-Source Robustness: Stain jittering, which introduces small random variations, helps models become more robust to variations within the same data source (e.g., same scanner/lab), leading to near-perfect performance on such data.
  • Trade-offs Exist: While stain jittering excels on data from the same source, stain normalization proves superior for generalizing to data from different sources (scanners and labs).

Key Results

  • No Stain Processing (NP): Achieved high F1 scores (0.947) on the Ventana validation set but dropped significantly on other datasets (e.g., 0.619 on Hamamatsu, 0.746 on Aperio 1, 0.727 on Aperio 2).
  • Stain Normalization (SN): Showed substantial improvements across all validation sets, with F1 scores of 0.982 (Ventana), 0.943 (Hamamatsu), 0.903 (Aperio 1), and 0.847 (Aperio 2).
  • Stain Jittering (SJ): Achieved a perfect F1 score (1.000) on the Ventana validation set but performed worse than stain normalization on datasets from different scanners/labs (0.939 on Hamamatsu, 0.814 on Aperio 1, 0.747 on Aperio 2).
  • The SN model consistently outperformed the SJ model when validating on data from different laboratories or scanners, highlighting its advantage for cross-dataset generalization.

Limitations and Future Work

  • Dataset Size: The study acknowledges that the training dataset size was limited, which might have influenced the performance differences between stain normalization and jittering. A smaller dataset might favor stain normalization as it constrains the learning domain.
  • Ground-Truth Noise: The authors recognize that the binary CD labels are based on subjective pathological diagnoses, which can introduce uncertainty or mislabeling.
  • Future Work: The researchers suggest using semi-supervised learning to mitigate ground-truth noise and improve model robustness by allowing the network to relabel outlier data. They also propose expanding the dataset size in future work.

Summary updated

Abstract

Around 1% of the population of the UK and North America have a diagnosis of coeliac disease (CD), due to a damaging immune response to the small intestine. Assessing whether a patient has CD relies primarily on the examination of a duodenal biopsy, an unavoidably subjective process with poor inter-observer concordance. Wei et al. [11] developed a neural network-based method for diagnosing CD using a dataset of duodenal biopsy whole slide images (WSIs). As all training and validation data came from one source, there was no guarantee that their results would generalize to WSIs obtained from different scanners and laboratories. In this study, the effects of applying stain normalization and jittering to the training data were compared. We trained a deep neural network on 331 WSIs obtained with a Ventana scanner (WSIs; CD: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>190</mml:mn></mml:math>; normal: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>141</mml:mn></mml:math>) to classify presence of CD. In order to test the effects of stain processing when validating on WSIs scanned on varying scanners and from varying laboratories, the neural network was validated on 4 datasets: WSIs of slides scanned on a Ventana scanner (WSIs; CD: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>48</mml:mn></mml:math>; normal: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>35</mml:mn></mml:math>), WSIs of the same slides rescanned on a Hamamatsu scanner (WSIs; CD: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>48</mml:mn></mml:math>; normal: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>35</mml:mn></mml:math>), WSIs of the same slides rescanned on an Aperio scanner (WSIs; CD: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>48</mml:mn></mml:math>; normal: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>35</mml:mn></mml:math>), and WSIs of different slides scanned on an Aperio scanner (WSIs; CD: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>38</mml:mn></mml:math>; normal: <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>37</mml:mn></mml:math>). Without stain processing, the F1 scores of the neural network were <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.947</mml:mn></mml:math>, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.619</mml:mn></mml:math>, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.746</mml:mn></mml:math><sub>,</sub> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.727</mml:mn></mml:math> when validating on the Ventana validation WSIs, Hamamatsu and Aperio rescans of the Ventana validation WSIs, and Aperio WSIs from a different source respectively. With stain normalization, the performance of the neural network improved significantly with respective F1 scores <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.982</mml:mn></mml:math>, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.943</mml:mn></mml:math>, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.903</mml:mn></mml:math>, and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.847</mml:mn></mml:math>. Stain jittering resulted in a better performance than stain normalization when validating on data from the same source F1 score <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>1.000</mml:mn></mml:math>, but resulted in poorer performance than stain normalization when validating on WSIs from different scanners (F1 scores <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.939</mml:mn></mml:math>, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.814</mml:mn></mml:math><sub>,</sub> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mn>0.747</mml:mn></mml:math>). This study shows the importance of stain processing, in particular stain normalization, when training machine learning models on duodenal biopsy WSIs to ensure generalizability between different scanners and laboratories.
BibTeX
@article{schreiber2023stain,
  author = {B.A. Schreiber and Jim Denholm and Julian Gilbey and Carola‐Bibiane Schönlieb and Elizabeth J. Soilleux},
  title = {{Stain normalization gives greater generalizability than stain jittering in neural network training for the classification of coeliac disease in duodenal biopsy whole slide images}},
  journal = {Journal of Pathology Informatics},
  year = {2023},
  doi = {10.1016/j.jpi.2023.100324},
}
Figure