Preserving Geometric Integrity in Image Translation

22 concepts22 papers1 proposal

Unpaired image-to-image translation has mastered the art of texture swapping. Standard generative models can effortlessly turn a sunny day into a rainy night or a horse into a zebra. They succeed because these transformations are primarily statistical shifts in color and local texture, leaving the underlying spatial layout intact.

However, when a transformation requires modifying the underlying shape—such as translating a cat into a human face, adapting a synthetic CAD model into a real-world object, or rendering a street view from a satellite image—standard models break down. Without explicit spatial constraints, purely data-driven networks suffer from geometric inconsistency. They introduce pixel-level artifacts, ghostly blurs, and structural hallucinations.

To bridge domains with large structural discrepancies, a growing body of research focuses on anchoring generative models with geometric and semantic constraints.

The Cost of Geometric Hallucination

Why do standard translation models warp shapes? Generative objectives like cycle consistency or standard adversarial losses penalize distribution mismatches, but they do not explicitly understand physical space. When forced to map between domains with different geometries, the network takes the path of least resistance, stretching and distorting object boundaries to satisfy the discriminator.

In many applications, this structural distortion is catastrophic. When generating synthetic data to train downstream perception models, geometric hallucinations invalidate the precise ground-truth labels generated by simulations:

To solve these issues, researchers have developed three core strategies: disentangling structure from appearance, utilizing explicit geometric blueprints, and enforcing physical or multi-hop constraints.

Disentangling What from How

One powerful intuition for preserving geometry is to split the latent space into two independent streams: one that captures what the object is shaped like (geometry) and one that captures how it looks (appearance).

By forcing this separation, the network can manipulate style without warping the underlying structural skeleton. In TransGaGa: Geometry-Aware Unsupervised Image-To-Image Translation, Wu et al. leverage this disentanglement to reshape objects across domains with massive structural differences, such as mapping a cat's face to a human's face, while preserving identity and style.

This mechanism also proves vital when cleaning or recovering images where the target structure must remain rigid. For instance, Zhao et al. demonstrate in Semi-Supervised Hand Appearance Recovery via Structure Disentanglement and Dual Adversarial Discrimination that isolating a hand's underlying structure using a Vision Transformer allows the model to strip away motion-capture markers and occlusions without distorting the hand's actual silhouette or pose.

[Input Image] ---> [Encoder] ---> Spatial Structural Code (Rigid)  ---\
                             ---> Style/Appearance Code (Fluid)  ---> [Generator] ---> [Output Image]

Explicit Structural Blueprints

Instead of relying on the network to implicitly discover geometry, another approach feeds the model an explicit structural "blueprint" to guide the generation process. This blueprint acts as an immutable spatial anchor.

For example, when changing a person's pose, directly transforming raw pixels often leads to blurred boundaries. Song et al. solve this in Unsupervised Person Image Generation With Semantic Parsing Transformation. by first predicting a discrete semantic map—a body part blueprint—for the new pose, and then painting textures onto that map.

Similarly, man-made environments demand rigid geometric constraints; a wobbly line instantly ruins architectural realism. To combat this, Xue et al. introduce the Neural Wireframe Renderer: Learning Wireframe to Image Translations, which forces the network to jointly reconstruct sparse 2D wireframes (lines and junctions) while generating the image, ensuring straight lines and sharp angles remain perfectly intact.

When explicit blueprints like wireframes are unavailable, pre-trained generative priors or attention mechanisms can serve as structural guides. Kim et al. demonstrate in U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation that attention mechanisms can highlight regions requiring significant shape modification. For even more distant categories (e.g., translating a bird into a car), Yang et al. show in Unsupervised Image-to-Image Translation with Generative Prior that leveraging the semantic knowledge embedded within pre-trained class-conditional GANs provides the necessary structural grounding to reconcile large geometric discrepancies.

Physical Projections and Multi-Hop Constraints

In cross-view synthesis—such as generating a ground-level street panorama from an overhead satellite image—the geometric transformation is too extreme for standard black-box models. These models frequently hallucinate buildings in physically impossible locations because they lack a concept of perspective.

To resolve this, researchers incorporate explicit 3D geometric relationships directly into the network architecture. In Geometry-Aware Satellite-to-Ground Image Synthesis for Urban Areas, Lu et al. enforce physical constraints to ensure synthesized skylines align with actual geographic footprints. Taking this further, Campbell et al. introduce a mathematical "Satellite to Street-view Projection" (S2SP) module in Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery, explicitly mapping overhead pixels to their corresponding ground-level coordinates before rendering the scene.

[Satellite View] ---> [3D Projection Module (S2SP)] ---> [Geometrically Grounded Layout] ---> [Refinement Generator] ---> [Street Panorama]

Alternatively, when the geometric gap is wide but cannot be modeled by a simple projection, the transformation can be broken down into smaller, manageable pieces. Lira et al. propose this incremental approach in GANHopper: Multi-hop GAN for Unsupervised Image-to-Image Translation. Instead of forcing a single generator to execute a massive geometric leap, GANHopper uses a single generator to execute a sequence of gradual "hops," enforcing local smoothness and intermediate hybrid states to preserve the original image's background and pose.

Finally, structural integrity can be maintained through specialized loss functions that penalize geometric deformation. Sigillo et al. employ a dual-flow generator alongside a structural dissimilarity loss in StawGAN: Structural-Aware Generative Adversarial Networks for Infrared Image Translation to preserve sharp vehicle contours when translating thermal infrared images to RGB. To ground depth estimation models without real-world depth labels, Rodriguez et al. introduce DESC in DESC: Domain Adaptation for Depth Estimation via Semantic Consistency, enforcing strict consistency between predicted depth maps and high-level semantic boundaries.

By moving away from unconstrained pixel manipulation and toward geometry-aware architectures, these methods ensure that image translation models respect the rigid laws of physical space.

Key Foundational Work

AugGAN: Cross Domain Adaptation with GAN-Based Data Augmentation (2018) — contributed 1 concept elements to this direction.
T ^2 2 Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks (2018) — contributed 1 concept elements to this direction.
SRDA: Generating Instance Segmentation Annotation via Scanning, Reasoning and Domain Adaptation (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_structural_inconsistency_in_domain_adapt_6897a789,
  title = {structural inconsistency in domain adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-inconsistency-in-domain-adaptation/rdr_6897a789da7e4b6b8d946ec8d504683f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_structural_inconsistency_in_domain_adapt_6897a789_v1,
  title = {structural inconsistency in domain adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-inconsistency-in-domain-adaptation/rdr_6897a789da7e4b6b8d946ec8d504683f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake