Conditional Generative Adversarial Refinement Networks for Unbalanced Medical Image Semantic Segmentation

Mina Rezaei, Haojin Yang, Konstantin Harmuth, Christoph MeinelWACV 2019 · OpenReview

Introduction

In medical image analysis, semantic segmentation often encounters the significant challenge of class imbalance. "Healthy" pixels typically outnumber "lesion" or "target" pixels by a massive margin. Standard deep learning models trained on such data often follow the path of least resistance, achieving high precision by correctly identifying healthy tissue but suffering from low sensitivity (recall) by missing actual disease markers. In clinical settings, a false negative—missing a tumor—is far more dangerous than a false positive. This paper introduces a novel architectural approach to solve this imbalance by treating error correction as a primary learning task.

Methodology

The authors propose a three-part ensemble architecture known as a Conditional Generative Refinement Adversarial Network (CGR-GAN). The framework consists of:

  1. Generative Network (G): A modified U-Net that performs the initial pixel-level segmentation.
  2. Discriminative Network (D): A standard GAN component that attempts to distinguish ground-truth masks from the synthetic masks produced by the generator.
  3. Refinement Network: The core innovation of this work. This separate U-Net-style architecture takes the generator's output and the original image to predict two specific error masks: a False Positive mask and a False Negative mask.
Overview of the proposed Conditional Generative Refinement Adversarial Network architecture, illustrating the flow from initial generation to adversarial discrimination and final refinement.

The final segmentation is calculated by taking the generator's output, subtracting the predicted false positives, and adding the predicted false negatives. To handle the 3D nature of medical volumes (like MRI or CT), the authors utilize Bidirectional LSTMs to capture spatial consistency between slices. Furthermore, Patient-wise Batch Normalization is employed to account for high variance in image intensity between different individuals.

Experimental Results

The model was evaluated on four diverse benchmarks: BraTS-2017 (brain tumors), LiTS-2017 (liver lesions), and two microscopic cell segmentation datasets (MDA231 and PhC-HeLa).

The refinement step provided a substantial performance boost across all datasets:

  • Liver Lesions (LiTS): The refinement network increased the Dice score by up to 9% compared to a standard cGAN.
  • Brain Tumors (BraTS): The model achieved a Dice score of 0.86 for the whole tumor, proving competitive with top-ranked challenge teams.
  • Cell Segmentation: On the MDA231 dataset, the method reached a segmentation accuracy of 0.93, outperforming standard U-Nets.
Qualitative segmentation results on sequential brain MRI slices from the BraTS dataset, showing the model's performance across different spatial contexts.

Discussion and Conclusion

The primary insight of this research is that treating error correction as a "first-class citizen" significantly cleans up the boundaries of segmented lesions. By explicitly asking a sub-network to identify where the primary network is likely to fail, the system achieves state-of-the-art results in highly unbalanced scenarios. While the refinement network adds computational overhead, the model remains efficient, capable of segmenting a full 155-slice MRI volume in under a minute. Future work will focus on expanding the network to handle multi-task learning, such as simultaneous disease classification and segmentation.

Summary updated

Abstract

We propose a new generative adversarial architecture to mitigate imbalance data problem in medical image semantic segmentation where the majority of pixels belongs to a healthy region and few belong to lesion or non-health region. A model trained with imbalanced data tends to bias towards healthy data which is not desired in clinical applications and predicted outputs by these networks have high precision and low sensitivity. We propose a new conditional generative refinement network with three components: a generative, a discriminative, and a refinement networks to mitigate imbalanced data problem through ensemble learning. The generative network learns to the segment at the pixel level by getting feedback from the discriminative network according to the true positive and true negative maps. On the other hand, the refinement network learns to predict the false positive and the false negative masks produced by the generative network that has significant value, especially in medical application. The final semantic segmentation masks are then composed by the output of the three networks. The proposed architecture shows state-of-the-art results on LiTS-2017 for simultaneous liver and lesion segmentation, and MDA231 for microscopic cell segmentation. We have achieved competitive results on BraTS-2017 for brain tumor segmentation.
BibTeX
@inproceedings{rezaei2019conditional,
  author = {Mina Rezaei and Haojin Yang and Konstantin Harmuth and Christoph Meinel},
  title = {{Conditional Generative Adversarial Refinement Networks for Unbalanced Medical Image Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=1Z2r-VSPHkE},
}

Loading related content...

Figure