Multimarginal Wasserstein Barycenter for Stain Normalization and Augmentation

Saad Nadeem, Travis Hollmann, Allen Tannenbaum

TL;DR

This paper introduces a mathematically robust framework for H&E stain normalization and augmentation using multimarginal Wasserstein barycenters. By treating color distributions as probability measures and finding their optimal "average" (barycenter), the authors can naturally incorporate multiple reference images to produce more realistic stain transfers and diverse data augmentations, achieving state-of-the-art results in nuclei segmentation tasks.

What Problem Does This Paper Address?

In digital pathology, Hematoxylin and Eosin (H&E) staining is the gold standard for visualizing tissue structures. However, the resulting images suffer from significant variability due to differing lab protocols, reagent manufacturers, and slide scanner hardware.

This "stain jitter" is a major hurdle for both human pathologists and computer-aided diagnosis (CAD) systems. For deep learning models, inconsistent staining leads to poor generalization; a model trained on data from one hospital might fail completely at another. While existing methods like StainGAN or sparse decomposition (Vahadane et al.) exist, they often require large amounts of paired training data or struggle to capture the full spectrum of realistic stain variations.

Methods

The authors leverage Optimal Mass Transport (OMT) theory. Specifically, they represent the color distribution of an image as a probability measure in the Lab color space.

The Wasserstein Barycenter

The core idea is the Wasserstein distance ($W_2$), which measures the "work" required to transform one distribution into another. A Wasserstein barycenter is the distribution that minimizes the sum of squared $W_2$ distances to a set of input distributions.

  • Traditional Approach ($N=2$): Interpolating between a source image and a single target reference. This creates a "geodesic" path in the distribution space, allowing for smooth stain transfer.
  • Multimarginal Approach ($N > 2$): The authors' primary contribution is extending this to multiple references. By using intermediate reference images, the model can navigate a broader spectrum of stain variations.
Fig. 2. Multimarginal Wasserstein Barycenter with 1, 2, and 3 reference images, i.e., N = 2, 3 or 4 in (6). The images with blue borders are input/source images, the ones with green are the (intermediate) references and the red ones are the final ref- erence/target images. Reference images can be from the same or different domains. Ideally in the multimarginal case, one of the intermediate references should have some background.

To make this computationally feasible, they employ the Sinkhorn algorithm with entropic regularization. This allows for fast, iterative calculation of the barycenter, making the method scalable to high-resolution pathology images.

Datasets

The method was evaluated on two primary benchmarks:

  1. MITOS-ATYPIA’14: Used to evaluate stain normalization. It contains tissue sections scanned by two different scanners (Aperio and Hamamatsu), providing a ground truth for cross-scanner normalization.
  2. MoNuSeg: A multi-organ nuclei segmentation dataset. This was used to test how well the proposed normalization and augmentation improve downstream task performance across different organs (Breast, Liver, Kidney, Prostate, etc.).

Key Insights and Ideas

  • Color Space Matters: The authors perform the transport in the Lab color space because its channels are decorrelated, which is more effective for color transfer than standard RGB.
  • Augmentation via Interpolation: Instead of just normalizing to a target, the authors use the points along the geodesic between distributions as augmented data. Because these points are mathematically derived from real tissue distributions, the resulting "fake" images are physically and biologically more realistic than simple random color shifts.
  • The Power of Intermediate References: Adding even one intermediate reference image significantly stabilizes the normalization. The authors found that as long as references contain varying amounts of background visibility, the resulting normalization is much more robust to outliers.
Fig. 3. Multimarginal Wasserstein Barycenter comparison with state of the art meth- ods on MITOS-ATYPIA’14 challenge dataset. The blue bordered image is the input image and the two red bordered images are the references (with the one above being the intermediate reference). m is the number of intermediate references (in traditional Wasserstein barycenter m = 0; there is no intermediate reference).

Key Results

The multimarginal approach ($m=1$, meaning one intermediate reference) outperformed several state-of-the-art methods, including StainGAN and Vahadane's method.

  • Normalization Accuracy: On the MITOS-ATYPIA dataset, the proposed method achieved a Structural Similarity Index (SSIM) of 0.73, significantly higher than StainGAN (0.68) and Vahadane (0.63).
  • Segmentation Performance: In the MoNuSeg challenge, combining their stain normalization (SN) and stain augmentation (SA) with a standard CNN architecture led to an Overall Aggregated Jaccard Index (AJI) of 0.5976, compared to the baseline of 0.5083.
  • Efficiency: While not the absolute fastest (Reinhard's simple color matching remains the speed leader), the Sinkhorn-based approach is efficient enough for practical use, processing 500 images in roughly 6 minutes on a single GPU.
Table 1. Stain Normalization Comparison (Mean ± Standard Deviation) on MITOS- ATYPIA’14 challenge dataset using Structural Similarity Index (SSIM) [20], Feature Similarity Index (FSIM) [21]. The time is the total time taken for normalizing all 500 images.

Limitations and Future Work

The authors note that the choice of reference images is important; ideally, at least one reference should have visible background to help the algorithm distinguish between tissue and empty space.

While the Sinkhorn algorithm is fast, the multimarginal case ($N > 4$) could become computationally expensive. Looking forward, the authors suggest integrating the Wasserstein barycenter formulation directly into the loss functions of deep neural networks, potentially allowing models to learn stain-invariance end-to-end.

Summary updated

Abstract

Variations in hematoxylin and eosin (H&E) stained images (due to clinical lab protocols, scanners, etc) directly impact the quality and accuracy of clinical diagnosis, and hence it is important to control for these variations for a reliable diagnosis. In this work, we present a new approach based on the multimarginal Wasserstein barycenter to normalize and augment H&E stained images given one or more references. Specifically, we provide a mathematically robust way of naturally incorporating additional images as intermediate references to drive stain normalization and augmentation simultaneously. The presented approach showed superior results quantitatively and qualitatively as compared to state-of-the-art methods for stain normalization. We further validated our stain normalization and augmentations in the nuclei segmentation task on a publicly available dataset, achieving state-of-the-art results against competing approaches.
BibTeX
@inproceedings{nadeem2020multimarginal,
  author = {Saad Nadeem and Travis Hollmann and Allen Tannenbaum},
  title = {{Multimarginal Wasserstein Barycenter for Stain Normalization and Augmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  doi = {10.1007/978-3-030-59722-1\_35},
}
Figure