Abstract
We address the problem of unpaired geometric image-to-image translation. Rather than transferring the style of an image as a whole, our goal is to translate the geometry of an object as depicted in different domains while preserving its appearance characteristics. Our model is trained in an unpaired fashion, i.e. without the need of paired images during training. It performs all steps of the shape transfer within a single model and without additional post-processing stages. Extensive experiments on the VITON, CMU-Multi-PIE and our own FashionStyle datasets show the effectiveness of the method. In addition, we show that despite their low-dimensionality, the features learned by our model are useful to the item retrieval task.
Introduction
Traditional image-to-image (I2I) translation models, such as CycleGAN, have demonstrated remarkable success in style transfer—changing the artistic texture of an image while maintaining its underlying geometry. However, many practical applications, such as virtual try-on, require the inverse: radical geometric transformation (e.g., warping a flat garment to fit a human body) while strictly preserving the original appearance and texture.
Existing solutions often rely on large-scale paired datasets, which are expensive and difficult to collect. This research introduces the Unpaired Shape Transforming (UST) method, a novel framework that learns complex geometric shifts between domains without requiring paired training examples. By decomposing images into shared style and domain-specific content spaces, UST enables high-fidelity shape translation across diverse datasets.
Proposed Methodology
The core of the UST framework is an Asymmetric Two-Stream Model. This architecture is designed to handle the inherent complexity difference between domains—for instance, a simple catalog image (Domain B) versus a complex image of a person wearing that item (Domain A).
Key Architectural Components
- Shared Latent Space: The model assumes that while the geometry (content) varies across domains, the appearance (style) can be represented in a shared space. This allows the model to extract a style code that remains invariant to the object's shape.
- The "Fit-in" Module: To resolve the ambiguity of how a flat object should deform onto a body, the "Fit-in" module merges clothing feature maps with target context information (such as a body mask). This guides the network toward a deterministic and realistic pose.
- Mask-Based Attention: An attention mechanism focuses the generator and discriminator on the object of interest, preventing background noise from degrading the translation quality.
- Hybrid Loss Functions: Training is stabilized using a combination of adversarial loss for realism, cycle-consistency loss for reversibility, and perceptual loss to ensure fine-grained texture preservation.
Experimental Results
The UST method was evaluated on the VITON benchmark, the CMU Multi-PIE face dataset, and a proprietary FashionStyle dataset. The model demonstrated superior performance in both "Try-on" (Domain B to A) and "Take-off" (Domain A to B) tasks.
Quantitative and Qualitative Performance
In the "take-off" task on the FashionStyle dataset, UST achieved an SSIM of 61.19 and an LPIPS score of 34.37, significantly outperforming baseline models like MUNIT (SSIM 45.97). Visually, UST successfully warps complex patterns like stripes and logos without the "melting" artifacts common in earlier GAN architectures.
Beyond translation, the learned style features proved highly effective for cross-domain retrieval. Despite using a compact 8-dimensional style code, the model achieved a 37.6% Top-5 recall rate when matching photos of people to their corresponding catalog items.
Conclusion and Future Work
The UST framework provides a robust solution for unpaired shape translation, effectively decoupling appearance from geometry. While the model currently relies on external masks and can struggle with heavy occlusions (e.g., crossed arms), it represents a significant step toward flexible, data-efficient virtual try-on systems. Future research will focus on integrating mask generation directly into the end-to-end pipeline to improve robustness against segmentation errors.
Summary updated