Style Consistent Image Generation for Nuclei Instance Segmentation

Xuan Gong, Shuyan Chen, Baochang Zhang, David S. Doermann

Abstract

Deep learning models for nuclei instance segmentation require large amounts of high-quality annotated data, which is expensive and time-consuming to produce. Furthermore, the domain gap between different organs and staining protocols often hinders model generalization. This paper introduces a framework that generates synthetic, style-consistent histopathology images by integrating a segmentation network directly into a Generative Adversarial Network (GAN) training loop. By utilizing instance deformation and style adaptation, the generator creates realistic training samples that specifically target the weaknesses of the segmentation model through adversarial feedback.

Introduction

The "data-hungry" nature of deep learning is a significant bottleneck in medical image analysis. Annotating a small set of nuclei patches can take an expert pathologist hundreds of hours. Beyond volume, the "domain gap"—variations in tissue types, stains, and imaging hardware—prevents models trained on one dataset from performing well on another. While standard GANs can generate visually realistic images, they often lack the task-specific utility needed for robust segmentation. This work addresses this gap by ensuring a feedback loop between image generation and the downstream segmentation task.

Proposed Architecture

The framework consists of three primary components integrated into a single adversarial loop: the Generator, the Discriminator, and the Segmentation Network.

Overview of the proposed framework, including Instance Deformation, Style Adaptation, and the Segmentation Net.

1. The Generator

The generator operates in two stages. First, Instance Deformation takes existing nuclei masks and applies random resizing or warping to create novel spatial arrangements. Second, Style Adaptation uses Adaptive Instance Normalization (AdaIN) to transfer the texture and color patterns (style) of real pathology images onto these deformed masks.

2. The Discriminator and Segmentation Net

A standard discriminator ensures visual realism by distinguishing between real and synthetic images. Simultaneously, a Segmentation Net (based on Mask R-CNN) provides adversarial feedback. Instead of just learning to segment, it acts as a critic for the generator, identifying synthetic images that are either too simplistic or unrepresentative of real-world challenges.

Adversarial Task Training

A core innovation of this work is the "min-max" game played between the generator and the segmentation network. The generator is optimized to synthesize images that are difficult for the segmentation network to process, effectively acting as an automated, task-aware data augmentation system.

The backpropagation flow showing how the segmentation loss ($L_S$) and discriminator loss ($L_D$) provide adversarial signals to the generator via gradient reversal.

By treating the segmentation loss as an adversarial signal, the generator learns to produce "hard examples" that force the segmentation network to become more robust and generalized. This ensures that the synthetic data directly improves the model's ability to handle diverse morphology and clear boundaries.

Experimental Results

The method was evaluated on three major datasets: Cell17, TCGA (MoNuSeg), and TNBC.

  • Performance: On the TCGA dataset, the model achieved an Aggregated Jaccard Index (AJI) of 0.6346, outperforming the previous state-of-the-art (0.6107).
  • Generalization: The model demonstrated strong performance on "unseen" organs (0.6613 AJI), indicating that the synthetic data helps the network learn generalizable features rather than memorizing specific textures.
  • Ablation: Studies confirmed that semantic supervision and focal loss in the segmentation branch significantly boosted Pixel-F1 and Dice scores compared to standard baselines.

Conclusion

This framework demonstrates that visual realism alone is insufficient for effective medical image augmentation. By incorporating task-specific adversarial feedback and style adaptation, the system generates synthetic data that significantly improves the performance and generalization of nuclei instance segmentation models. Future work will focus on reducing pre-training requirements and exploring purely generative mask creation.

Summary updated

Abstract

In medical image analysis, one limitation of the application of machine learning is the insufficient amount of data with detailed annotation, due primarily to high cost. Another impediment is the domain gap observed between images from different organs and different collections. The differences are even more challenging for the nuclei instance segmentation, where images have significant nuclei stain distribution variations and complex pleomorphisms (sizes and shapes). In this work, we generate style consistent histopathology images for nuclei instance segmentation. We set up a novel instance segmentation framework that integrates a generator and discriminator into the segmentation pipeline with adversarial training to generalize nuclei instances and texture patterns. A segmentation net detects and segments both real nuclei and synthetic nuclei and provides feedback so that the generator can synthesize images that can boost the segmentation performance. Experimental results on three public nuclei datasets indicate that our proposed method outperforms previous nuclei segmentation methods.
BibTeX
@inproceedings{gong2021style,
  author = {Xuan Gong and Shuyan Chen and Baochang Zhang and David S. Doermann},
  title = {{Style Consistent Image Generation for Nuclei Instance Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00404},
}
Figure