Multimarginal Wasserstein Barycenter for Stain Normalization and Augmentation

Saad Nadeem, Travis J. Hollmann, Allen R. Tannenbaum

TL;DR

This paper introduces a novel approach for normalizing and augmenting histology images using a multimarginal Wasserstein barycenter. By leveraging multiple reference images, their method achieves superior stain normalization and improves nuclei segmentation performance, outperforming existing state-of-the-art techniques.

What Problem Does This Paper Address?

Histology images, crucial for medical diagnosis, are often plagued by variations in staining. These inconsistencies arise from differences in lab protocols, scanners, and tissue preparation, directly impacting diagnostic accuracy and the performance of computer-aided diagnosis systems. While stain normalization aims to standardize these images, and augmentation can improve model robustness, existing methods often fall short, particularly for tasks like nuclei segmentation where deep learning approaches struggle with limited training data. This paper tackles the challenge of creating a robust and versatile method for both stain normalization and augmentation that can effectively handle these variations and improve downstream tasks.

Methods

The core of this paper is the application of the multimarginal Wasserstein barycenter for stain normalization and augmentation. The process involves:

  1. Color Space Conversion: Images are converted from RGB to the Lab color space, which is known to be effective for color transfer due to its decorrelated color channels.
  2. Wasserstein Barycenter: The authors first introduce the traditional Wasserstein barycenter (using one source and one reference image) for stain normalization. This involves interpolating the color distribution of the source image towards the reference distribution in the Lab color space.
  3. Multimarginal Extension: To overcome the limitations of the traditional approach, they extend it to a multimarginal Wasserstein barycenter. This allows for the incorporation of one or more intermediate reference images in addition to the primary reference. This is achieved by minimizing a functional that considers the Wasserstein distance to multiple distributions simultaneously. The multimarginal setting allows for a more nuanced and comprehensive interpolation, capturing a broader spectrum of stain variations.
  4. Computational Approach: For computing the barycenters, they employ a subgradient descent framework based on an entropic regularized Wasserstein distance, utilizing the Sinkhorn algorithm for efficient computation.

The novelty lies in the application of the multimarginal Wasserstein barycenter to this specific problem, a technique that has seen limited use in computer vision and medical imaging communities.

Datasets

  • MITOS-ATYPIA'14 challenge dataset: Used for evaluating stain normalization. This dataset consists of H&E stained tissue sections scanned by two different scanners, providing a realistic scenario for testing normalization methods.
  • MoNuSeg challenge dataset: Used for evaluating nuclei segmentation performance. This dataset is standard for nuclei segmentation tasks.

Key Insights and Ideas

  • Intermediate References are Key: The most significant insight is that incorporating intermediate reference images in the multimarginal setting dramatically improves stain normalization. These references can span different amounts of background visibility, allowing for a more robust and realistic interpolation.
  • Augmentation via Interpolation: The method naturally provides augmentation by generating intermediate images that represent a spectrum of stain variations between the source and reference(s). This is crucial for tasks where deep learning models struggle with limited diverse training data.
  • Color Distribution Space Matters: Performing normalization in the Lab color space, rather than RGB, is a standard but effective choice that decorrelates color channels and facilitates better color transfer.
  • Multimarginal Barycenter for Robustness: The multimarginal approach is more robust than traditional methods because it can better capture the complex variations in H&E staining by considering multiple reference distributions.

Key Results

  • Stain Normalization: The proposed method, particularly with one intermediate reference (m=1), achieved state-of-the-art results on the MITOS-ATYPIA'14 dataset, outperforming established methods like Reinhard et al., Macenko et al., Khan et al., Vahadane et al., and StainGAN in terms of Structural Similarity Index (SSIM) and Feature Similarity Index (FSIM). Specifically, the m=1 (Ours) achieved an SSIM of 0.73±0.06 and FSIM of 0.75±0.11, significantly higher than other methods.
  • Nuclei Segmentation: When applied to the MoNuSeg challenge dataset, the stain normalization and augmentation approach (C+G+SN+SA) also achieved state-of-the-art results, outperforming other combinations of normalization and augmentation techniques. The overall Aggregated Jaccard Index was 0.5976, the highest among the tested methods.
  • Scalability: The method is designed to be easily scalable to whole slide images as normalization is performed in the color distribution space and is not dependent on the number of pixels.

Limitations and Future Work

  • Computational Cost: While the authors employ efficient algorithms, computing Wasserstein barycenters can still be computationally intensive, especially for very large datasets or complex multimarginal settings.
  • Deep Learning Integration: The authors suggest exploring the Wasserstein barycenter formulation as a deep learning loss function in the future. This could potentially lead to more end-to-end trainable models that inherently handle stain variations.
  • Further Exploration of Augmentation: While the method naturally provides augmentation, further research could focus on optimizing the selection and use of intermediate references for more targeted augmentation strategies.

Summary updated

Abstract

Variations in hematoxylin and eosin (H&E) stained images (due to clinical lab protocols, scanners, etc) directly impact the quality and accuracy of clinical diagnosis, and hence it is important to control for these variations for a reliable diagnosis. In this work, we present a new approach based on the multimarginal Wasserstein barycenter to normalize and augment H&E stained images given one or more references. Specifically, we provide a mathematically robust way of naturally incorporating additional images as intermediate references to drive stain normalization and augmentation simultaneously. The presented approach showed superior results quantitatively and qualitatively as compared to state-of-the-art methods for stain normalization. We further validated our stain normalization and augmentations in the nuclei segmentation task on a publicly available dataset, achieving state-of-the-art results against competing approaches.
BibTeX
@inproceedings{nadeem2020multimarginal,
  author = {Saad Nadeem and Travis J. Hollmann and Allen R. Tannenbaum},
  title = {{Multimarginal Wasserstein Barycenter for Stain Normalization and Augmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  doi = {10.1007/978-3-030-59722-1\_35},
}
Figure