Semantic Aware Data Augmentation for Cell Nuclei Microscopical Images with Artificial Neural Networks

Alireza Naghizadeh, Hongye Xu, Mohab Mohamed, Dimitris N. Metaxas, Dongfang LiuICCV 2021 · OpenReview

Abstract

There exists many powerful architectures for object detection and semantic segmentation of both biomedical and natural images. However, a difficulty arises in the ability to create training datasets that are large and well-varied. The importance of this subject is nested in the amount of training data that artificial neural networks need to accurately identify and segment objects in images and the infeasibility of acquiring a sufficient dataset within the biomedical field. This paper introduces a new data augmentation method that generates artificial cell nuclei microscopical images along with their correct semantic segmentation labels. Data augmentation provides a step toward accessing higher generalization capabilities of artificial neural networks. An initial set of segmentation objects is used with Greedy AutoAugment to find the strongest performing augmentation policies. The found policies and the initial set of segmentation objects are then used in the creation of the final artificial images. When comparing the state-of-the-art data augmentation methods with the proposed method, the proposed method is shown to consistently outperform current solutions in the generation of nuclei microscopical images.

Introduction

Training deep neural networks for medical tasks—like identifying and segmenting cell nuclei—requires massive amounts of labeled data. However, acquiring "ground truth" labels in biology is a significant bottleneck: it requires expert knowledge, is prone to human error, and is incredibly time-consuming. While Generative Adversarial Networks (GANs) are often used to synthesize more data, they typically require large datasets themselves and fail to automatically generate the corresponding pixel-perfect segmentation masks needed for training.

This paper introduces a "semantic-aware" data augmentation framework designed specifically for cell nuclei microscopy. By combining expert-segmented cell objects with an automated augmentation policy search, the researchers can generate high-fidelity synthetic images complete with perfect segmentation masks, consistently outperforming state-of-the-art GANs on small biomedical datasets.

Methodology

The proposed pipeline moves away from "end-to-end" generation and toward a sophisticated "compositional" approach involving three main stages:

The process starts with a small set of "seed" objects (single cells and multi-cell clusters) segmented by experts. To ensure augmented cells still look "real," the authors use a Wasserstein AutoEncoder (WAE). The WAE is trained to reconstruct original cells. If an augmentation (like a specific rotation or color shift) makes a cell look "unnatural" to the AutoEncoder, that policy is ranked lower. A greedy search algorithm then finds the best combination of 11 different augmentation techniques.

The Autoencoder architecture used to evaluate the naturalness of segmented objects during the policy search phase.

2. Image Synthesis

Once the best policies are found, the system creates new images by "pasting" these augmented cells onto a background. Crucially, it mimics the distribution of cells (how many single vs. multi-cell clusters) found in the original data. Because the system knows exactly where it placed each cell, it generates the segmentation mask simultaneously.

The full semantic-aware data augmentation pipeline, illustrating the flow from original image extraction to the generation of artificial images and masks.

Datasets and Results

The authors evaluated their method on two primary datasets: the CAR-T Cell Dataset (156 high-resolution fluorescent microscopy images) and the Kaggle 2018 Data Science Bowl (589 images).

Image Quality and Downstream Performance

The "Sem-Aware" method achieved significantly lower Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) scores compared to StyleGAN2-Diff and BigGAN-Diff. For example, on the CAR-T dataset, the proposed method hit an FID of 8.37, while the best GAN reached only 31.32.

In terms of utility, adding 100 augmented images to training sets yielded performance boosts across 11 different state-of-the-art detection algorithms. In multi-organ nuclei segmentation tasks, the method improved Jaccard scores by over 10% on CAR-T and 25% on Kaggle data compared to baseline training.

Conclusion

The semantic-aware approach demonstrates that composition is often superior to hallucination for microscopy data. By leveraging the fact that these images are composed of distinct objects on simple backgrounds, the method overcomes the data hunger and label blindness of traditional GANs. While the method currently depends on an initial set of expert-segmented seeds, it provides a robust path toward higher generalization in biomedical imaging where data is scarce.

Summary updated

Abstract

There exists many powerful architectures for object detection and semantic segmentation of both biomedical and natural images. However, a difficulty arises in the ability to create training datasets that are large and well-varied. The importance of this subject is nested in the amount of training data that artificial neural networks need to accurately identify and segment objects in images and the infeasibility of acquiring a sufficient dataset within the biomedical field. This paper introduces a new data augmentation method that generates artificial cell nuclei microscopical images along with their correct semantic segmentation labels. Data augmentation provides a step toward accessing higher generalization capabilities of artificial neural networks. An initial set of segmentation objects is used with Greedy AutoAugment to find the strongest performing augmentation policies. The found policies and the initial set of segmentation objects are then used in the creation of the final artificial images. When comparing the state-of-the-art data augmentation methods with the proposed method, the proposed method is shown to consistently outperform current solutions in the generation of nuclei microscopical images.
BibTeX
@inproceedings{naghizadeh2021semantic,
  author = {Alireza Naghizadeh and Hongye Xu and Mohab Mohamed and Dimitris N. Metaxas and Dongfang Liu},
  title = {{Semantic Aware Data Augmentation for Cell Nuclei Microscopical Images with Artificial Neural Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=YtpGJPL4Im},
}
Figure