Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization

Mengwei Ren, Neel Dey, James Fishbaugh, Guido Gerig

Abstract

Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.

Introduction

Medical imaging studies often aggregate data from multiple centers to increase sample size and statistical power. However, images acquired from different scanners, even with similar protocols, exhibit significant "non-biological variability" in contrast, resolution, and noise. These "batch effects" can severely degrade the performance of downstream automated analysis tools. While unsupervised image-to-image translation methods like CycleGAN have been used to harmonize these datasets, they often suffer from instability and can introduce clinical artifacts, such as hallucinating or removing pathologies like tumors or lesions.

Methodology

The core contribution of this work is a framework that leverages anatomical shape as an invariant anchor during translation. Unlike standard GANs that operate primarily on image intensity, the proposed model treats morphological structure as the ground truth that must be preserved across domains.

The proposed segmentation-renormalized framework for unpaired image harmonization, showing both forward and backward translation cycles.

The architecture consists of three main components:

  1. Joint Segmentation and Semantic Extraction: The model simultaneously learns to segment anatomical structures. These masks are processed by a semantic extractor to create a "structural embedding."
  2. Segmentation-Renormalization: Instead of standard normalization layers, the generator uses Segmentation Renormalized ResBlocks. These blocks employ Feature-wise Linear Modulation (FiLM) to adjust feature maps based on the structural embedding.
  3. Multi-Task Objective: The network is optimized using a combination of adversarial, cycle-consistency, and segmentation losses.
Comparison between a standard ResBlock and the proposed Segmentation Renormalized ResBlock using FiLM modulation.

By injecting structural information deep into the network's layers, the model prevents the "washing away" of spatial context and discourages steganographic mapping—a common issue where GANs hide source information in high-frequency noise.

Experimental Setup

The model was validated on three diverse datasets:

  • MS-SEG: FLAIR MRI scans of Multiple Sclerosis patients (1.5T vs 3T).
  • RETOUCH: Retinal OCT scans from different manufacturers (Cirrus vs Spectralis).
  • IXI: T1w MRI of healthy subjects (GE 1.5T vs Philips 3T).

Results and Discussion

The proposed method demonstrated superior performance across multiple metrics. It achieved lower Frechet Inception Distance (FID) and Kernel Inception Distance (KID), indicating higher visual fidelity to the target domain. Critically, it improved downstream diagnostic utility; for example, on the MS-SEG dataset, lesion segmentation accuracy (Dice coefficient) improved from 0.42 with standard CycleGAN to 0.50 with the proposed method. Furthermore, the model showed enhanced robustness against self-adversarial attacks and noise perturbations compared to baseline architectures.

Conclusion

Segmentation-renormalized feature modulation provides a robust and clinically reliable approach to medical image harmonization. By anchoring the translation process in anatomical structure, the framework ensures that critical clinical features are preserved while effectively reducing inter-scanner heterogeneity.

Summary updated

Abstract

Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.
BibTeX
@article{ren2021segmentation,
  author = {Mengwei Ren and Neel Dey and James Fishbaugh and Guido Gerig},
  title = {{Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  url = {https://openreview.net/forum?id=dQ9QWSxLS9i},
}
Figure