Generative adversarial networks (GANs) and diffusion models excel at altering the style of an image—such as turning a horse into a zebra or a summer landscape into a snowy winter scene. In these natural image tasks, the exact spatial coordinates of a leaf or a stripe do not matter, as long as the final output looks plausible to the human eye.
However, when translating a medical image—such as converting a magnetic resonance image (MRI) into a synthetic computed tomography (CT) scan, or digitally transforming a standard H&E stained pathology slide into an immunohistochemistry (IHC) stain—global aesthetic realism is secondary. The primary requirement is that the underlying biological anatomy must remain perfectly intact.
Standard unpaired image-to-image translation frameworks often fail this requirement. Because they are driven primarily by discriminators that evaluate distribution-level realism, they suffer from structural drift, boundary shifts, and anatomical hallucinations. A model might warp the shape of an organ or erase a critical blood vessel simply because doing so makes the contrast patterns match the target domain more easily. To make generative models reliable for clinical applications like automated diagnostics, surgical navigation, or data augmentation, researchers must explicitly force the networks to prioritize structural preservation over unconstrained stylistic realism.
The Root of the Problem: The Many-to-Many Mapping Trap
Why do standard generative models warp anatomy? In an unpaired translation setting, a model learns to map an image from domain $X$ to domain $Y$ without pixel-aligned training pairs. The discriminator's only job is to determine whether the generated image $G(x)$ looks like it belongs to domain $Y$.
This setup creates an under-constrained mathematical problem. There are infinitely many ways to rearrange the pixels of an input image $x$ to satisfy the target domain's distribution. Even when frameworks introduce a cycle-consistency loss ($F(G(x)) \approx x$) to ensure the image can be translated back to its source, the network can "cheat" by hiding adversarial information in high-frequency textures rather than maintaining true geometric invariance. As noted by Mao in Generative adversarial networks with adaptive normalization for synthesizing T2-weighted magnetic resonance images from diffusion-weighted images, unconstrained generative models are highly prone to these geometric deformations and organ warping.
To solve this, modern architectures introduce specific regularizers, structural priors, and architectural constraints that anchor the geometry while allowing the style to change.
Mechanism 1: Disentangling Content from Style
An intuitive way to protect anatomy is to force the network to separate what is in the image (the structural content) from how it is rendered (the stylistic contrast or stain). By decoupling these attributes into orthogonal pathways, changes in color and intensity cannot easily distort the underlying shapes.
Input Image ---> [ Content Encoder ] ---> Structural Features ---\
---> [ Generator ] ---> Translated Image
Target Style --> [ Style Encoder ] ---> Stylistic Features ---/
In virtual staining for digital pathology, failing to isolate these factors means that changes in stain intensity can inadvertently alter the perceived arrangement of tissue cells. To prevent this, Wang et al. developed ODA-GAN: Orthogonal Decoupling Alignment GAN Assisted by Weakly-supervised Learning for Virtual Immunohistochemistry Staining, which explicitly decouples tissue morphology from staining information to preserve pathological accuracy.
A similar strategy applies to cross-modality translation between entirely different scanning technologies (like MRI and CT). In A New Bidirectional Unsupervised Domain Adaptation Segmentation Framework, Ning et al. utilized a domain-aware pattern encoder to separate style from content, ensuring that the underlying anatomy remains identical across bidirectional translations.
Mechanism 2: Structural and Perceptual Constraints
Instead of relying purely on adversarial feedback, models can be regularized using loss functions designed to measure geometric and semantic consistency across domains.
Multi-Scale Structural Similarity
Rather than comparing raw pixel values—which change completely between modalities—models can optimize for structural relationships, such as local gradients and edge alignments. Venkatesh et al. demonstrated in Exploring semantic consistency in unpaired image translation to generate data for surgical applications that integrating multi-scale structural similarity acts as a powerful regularizer. This constraint prevents the model from misapplying textures or hallucinating features in the wrong locations, preserving the precise boundaries of surgical instruments and organs.
Feature-Level Perception Loss
Another approach evaluates images through the lens of a pre-trained deep network. By comparing the intermediate feature maps of the source image and the generated image, the model optimizes for high-level semantic patterns rather than raw pixel intensities. In MultiPathGAN: Structure Preserving Stain Normalization using Unsupervised Multi-domain Adversarial Network with Perception Loss, Nazki et al. showed that feature-level perception loss suppresses artifact creation and maintains the integrity of fine-grained anatomical structures during stain normalization.
Modality-Independent Structural Metrics
Certain mathematical descriptors are inherently invariant to contrast changes. For example, the Modality-Independent Neighborhood Descriptor (MIND) evaluates local self-similarity patches. In Unpaired Brain MR-to-CT Synthesis using a Structure-Constrained CycleGAN, the authors incorporated a structural loss based on these descriptors, successfully mitigating structural drift and achieving high spatial fidelity without paired data.
Mechanism 3: Architectural Biases and Attention
Sometimes, the best way to enforce structural preservation is to build it directly into the network's internal routing mechanisms.
Strict One-to-One Bijective Mappings
Standard CycleGAN frameworks employ two independent generator networks (one for $X \to Y$ and another for $Y \to X$), leaving room for loose, many-to-many approximations. To enforce a stricter geometric bond, Shen et al. designed One-to-one Mapping for Unpaired Image-to-image Translation. By utilizing a single, self-inverse neural network to handle both forward and backward translations, the architecture guarantees a strict mathematical bijection, preserving unique patient-specific details across domains.
Structural Attention
Transformers offer powerful global context, but they can inadvertently smear or delete complex, fine-grained cavities when translating across modalities. To counteract this, Xie introduced Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis. This architecture injects explicit structural knowledge (derived from foundational models like the Segment-Anything Model) directly into the transformer's attention layers, forcing the network to prioritize the geometric integrity of intricate biological structures like nasal cavities or joints.
Mechanism 4: Source Conditioning in Diffusion Models
As the state-of-the-art in generative modeling shifts from GANs to Denoising Diffusion Probabilistic Models (DDPMs), researchers have adapted structural preservation techniques to iterative denoising processes.
Unconstrained diffusion models can drift significantly over multiple denoising steps, creating plausible-looking but completely hallucinated anatomy. To anchor the generation, the original source image can be injected directly into the step-by-step denoising loop.
Source Image (Clean Anatomy)
|
v
[ Noisy Target Image ] ---> [ Denoising Step ] ---> [ Refined Target Image ]
In the context of MRI scanner harmonization, Durrer et al. demonstrated this mechanism in Diffusion Models for Contrast Harmonization of Magnetic Resonance Images. By concatenating the original source image with the noisy target image at each stage of the diffusion process, the source image acts as an unyielding structural guide. The model is restricted to modifying only the stylistic contrast and noise characteristics, ensuring that the patient's true underlying anatomy remains perfectly unaltered.
Key Foundational Work
Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation (2018) — contributed 2 concept elements to this direction.