An Unpaired Shape Transforming Method for Image Translation and Cross-Domain Retrieval

Kaili Wang, Liqian Ma, José Oramas, Luc Van Gool, Tinne Tuytelaars

Abstract

We address the problem of unpaired geometric image-to-image translation. Rather than transferring the style of an image as a whole, our goal is to translate the geometry of an object as depicted in different domains while preserving its appearance characteristics. Our model is trained in an unpaired fashion, i.e. without the need of paired images during training. It performs all steps of the shape transfer within a single model and without additional post-processing stages. Extensive experiments on the VITON, CMU-Multi-PIE and our own FashionStyle datasets show the effectiveness of the method. In addition, we show that despite their low-dimensionality, the features learned by our model are useful to the item retrieval task.

Introduction

Most image-to-image (I2I) translation methods, such as CycleGAN, excel at "style transfer"—altering textures or colors while maintaining identical geometry. However, many real-world applications, like virtual try-on, require shape transfer, where the object's geometry must be significantly transformed. Traditionally, training such models requires expensive "paired data" (e.g., the same person wearing the same item in the same pose). This paper introduces the Unpaired Shape Transforming (UST) framework, which performs complex geometric transformations using two independent, unpaired sets of images.

Methodology

The UST architecture is an asymmetric two-stream Generative Adversarial Network (GAN) that decomposes images into a shared content space and a shared style space.

The UST architecture featuring asymmetric streams, the Fit-in module, and attention mechanisms.

Key components include:

  • Asymmetric Streams: The model handles "Try-on" (Object-centric $\to$ Contextualized) and "Take-off" (Contextualized $\to$ Object-centric) directions.
  • The "Fit-in" Module: To address the "one-to-many" mapping problem, a reference "context" image (e.g., a person's silhouette) guides the transformation. The Fit-in module aligns feature maps to this mask, ensuring the generated object fits the target geometry.
  • Attention Mechanism: Shape masks ($m_A$) focus the network on the object of interest, preventing background or skin tone interference.
  • Hybrid Loss Functions: Optimization utilizes adversarial loss, cycle-consistency, perceptual loss (via VGG16), and a symmetry loss for natural-looking "take-off" results.

Experiments and Results

The model was evaluated on the VITON benchmark, the CMU-Multi-PIE face dataset, and a custom FashionStyle dataset.

Qualitative Performance

Compared to baselines like CycleGAN and MUNIT, the UST method produces significantly sharper images with fewer geometric artifacts. It successfully preserves intricate details like logos and patterns even after significant warping.

Qualitative comparison of the UST method (Ours) against CycleGAN and MUNIT for try-on and take-off tasks.

Quantitative and Retrieval Insights

On the FashionStyle dataset, UST achieved a mean SSIM of 61.19 for the "take-off" task, outperforming MUNIT (45.97) and CycleGAN (45.63). Furthermore, the learned style features, even at a low dimensionality (e.g., 8D), proved powerful for cross-domain retrieval, outperforming standard ResNet-50 features in finding catalog items from photos of people wearing them.

Conclusion and Future Work

The UST framework demonstrates that joint training of "try-on" and "take-off" tasks, combined with explicit geometric guidance, enables high-quality unpaired shape transfer. While the model is currently dependent on mask accuracy and can struggle with heavy occlusions, future work integrating 3D human modeling or advanced pose estimation holds promise for addressing these limitations.

Summary updated

Abstract

We address the problem of unpaired geometric image-to-image translation. Rather than transferring the style of an image as a whole, our goal is to translate the geometry of an object as depicted in different domains while preserving its appearance characteristics. Our model is trained in an unpaired fashion, i.e. without the need of paired images during training. It performs all steps of the shape transfer within a single model and without additional post-processing stages. Extensive experiments on the VITON, CMU-Multi-PIE and our own FashionStyle datasets show the effectiveness of the method. In addition, we show that despite their low-dimensionality, the features learned by our model are useful to the item retrieval task.
BibTeX
@misc{wang2018unpaired,
  author = {Kaili Wang and Liqian Ma and José Oramas and Luc Van Gool and Tinne Tuytelaars},
  title = {{An Unpaired Shape Transforming Method for Image Translation and Cross-Domain Retrieval}},
  year = {2018},
  eprint = {1812.02134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.02134},
}
Figure