Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization

Mengwei Ren, Neel Dey, James Fishbaugh, Guido Gerig

This blog-style summary provides an overview of the paper "Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization," published in IEEE Transactions on Medical Imaging.

TL;DR

Medical imaging studies often struggle with "batch effects" caused by different scanners, but standard AI tools for fixing this (like CycleGAN) often hallucinate or delete critical pathologies. This paper introduces a "segmentation-renormalized" framework that uses anatomical shapes to guide the image translation process at every layer of the neural network. The result is a more stable harmonization process that preserves clinical features like lesions while significantly improving the accuracy of downstream diagnostic tools.

What Problem Does This Paper Address?

In large-scale medical studies, researchers often aggregate data from multiple hospitals. However, a brain scan from a 1.5T Siemens scanner looks fundamentally different from one taken on a 3T Philips scanner—variations in contrast, noise, and resolution create "non-biological variability."

To fix this, researchers use unpaired image-to-image translation (like CycleGAN) to make images from "Site A" look like they came from "Site B." The problem? These models are notoriously "unconstrained." They might "cheat" by hiding information in high-frequency noise (steganography), or worse, they might invert the contrast of a tumor or remove a lesion entirely to satisfy the mathematical requirements of the GAN. In a clinical setting, these artifacts are unacceptable.

Methods

The authors propose a novel architecture that treats anatomical shape as the "ground truth" that must remain constant, even if the image's "style" changes.

  1. Joint Segmentation: Alongside the image generator, the model learns to predict anatomical segmentation masks (e.g., identifying grey matter, white matter, or lesions).
  2. Semantic Extractor: This predicted mask is passed through a sub-network to create a "structural embedding"—a mathematical representation of the patient's anatomy.
  3. Segmentation-Renormalization (The Core Innovation): Instead of using standard normalization layers (which can "wash away" spatial context), the authors use Feature-wise Linear Modulation (FiLM). The structural embedding is used to calculate specific scale ($\gamma$) and shift ($\beta$) parameters that modulate the neural network's features at every level of the generator.
  4. Multi-Task Learning: The system is trained end-to-end with a combination of adversarial losses (to match the target site's look), cycle-consistency losses (to ensure the translation is reversible), and segmentation losses (to ensure the anatomy stays the same).

Datasets

The model was evaluated across three distinct imaging modalities and challenges:

  • MS-SEG: FLAIR MRI scans of Multiple Sclerosis patients from three different scanners (Siemens 1.5T, Philips 3T, Siemens 3T). This tested the model's ability to preserve small, sparse lesions.
  • RETOUCH: Retinal Optical Coherence Tomography (OCT) from two different manufacturers (Cirrus vs. Spectralis). OCT has high inter-device variability, making this a difficult test for visual fidelity.
  • IXI: T1-weighted MRI of healthy subjects from two sites (GE 1.5T vs. Philips 3T), used to test general tissue harmonization.

Key Insights and Ideas

  • Shape as an Invariant: The central "aha" moment is that while intensities change between scanners, the underlying anatomy does not. By forcing the network to "renormalize" its features based on a segmentation map, the authors provide a structural "anchor" that prevents the GAN from drifting into hallucination.
  • Feature-Level vs. Image-Level: Most previous attempts at using segmentation only applied it as a loss function at the very end (the output). This paper shows that injecting structural information deep inside the network's internal layers (feature-level) is much more effective.
  • Defeating "Steganography": Standard CycleGANs often hide "source" information in invisible noise patterns to help them reconstruct the original image later. This paper demonstrates that their structural constraint makes the network more robust against this type of "cheating," leading to cleaner, more reliable images.

Key Results

The proposed method consistently outperformed standard CycleGAN and other segmentation-aware baselines:

  • Visual Fidelity: The model achieved significantly lower Kernel Inception Distance (KID) and Frechet Inception Distance (FID), meaning the harmonized images were statistically much closer to real target-site images.
  • Downstream Accuracy: On the MS-SEG dataset, using the authors' harmonized images improved the Dice Coefficient (a measure of segmentation overlap) for lesion detection from 0.42 (standard CycleGAN) to 0.50.
  • Robustness: The model was far more stable when faced with "self-adversarial attacks" or random noise. While standard models saw their reconstructions collapse under slight perturbations, the segmentation-renormalized model maintained structural integrity.

Limitations and Future Work

  • Demographic Assumptions: The model assumes that the patient populations at different sites are roughly similar. It does not yet account for "biological batch effects," such as one site having significantly older patients than another.
  • 2D vs. 3D: The current framework operates on 2D slices. While this makes it versatile (especially for anisotropic data like OCT), it can lead to slight inconsistencies when looking at a 3D volume slice-by-slice.
  • Label Dependency: While the model can use "pseudo-labels" (algorithmically generated masks), it still performs best when some high-quality segmentation labels are available for training. Future work may look into more "label-efficient" versions of this architecture.

Summary updated

Abstract

Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.
BibTeX
@article{ren2021segmentation,
  author = {Mengwei Ren and Neel Dey and James Fishbaugh and Guido Gerig},
  title = {{Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  url = {https://openreview.net/forum?id=dQ9QWSxLS9i},
}

Loading related content...

Figure