TailorGAN: Making User-Defined Fashion Designs

Lele Chen, Justin Tian, Guo Li, Cheng-Haw Wu, Erh-Kan King, Kuan-Ting Chen, Shao-Hang Hsieh, Chenliang Xu

Abstract

Attribute editing has become an important and emerging topic of computer vision. In this paper, we consider a task: given a reference garment image A and another image B with target attribute (collar/sleeve), generate a photorealistic image which combines the texture from reference A and the new attribute from reference B. The highly convoluted attributes and the lack of paired data are the main challenges to the task. To overcome those limitations, we propose a novel self-supervised model to synthesize garment images with disentangled attributes (e.g., collar and sleeves) without paired data. Our method consists of a reconstruction learning step and an adversarial learning step. The model learns texture and location information through reconstruction learning. And, the model's capability is generalized to achieve single-attribute manipulation by adversarial learning. Meanwhile, we compose a new dataset, named GarmentSet, with annotation of landmarks of collars and sleeves on clean garment images. Extensive experiments on this dataset and real-world samples demonstrate that our method can synthesize much better results than the state-of-the-art methods in both quantitative and qualitative comparisons.

Introduction

In the world of AI-driven fashion, "attribute editing" is a significant challenge. Imagine taking a floral print shirt and instantly seeing how it would look with a Peter Pan collar versus a lapel. TailorGAN is a novel generative framework designed to solve this by allowing users to swap specific fashion attributes—like changing a round collar to a V-neck or short sleeves to long sleeves—while seamlessly preserving the garment's original texture.

The TailorGAN task: combining the texture of one garment with the structural attribute (collar or sleeve) of another.

The primary technical hurdle is entanglement, where texture (pattern) and structure (shape) are deeply intertwined in generative models. Standard tools like CycleGAN often inadvertently change the color of the entire garment or blur the fabric pattern when attempting to modify a single attribute. Furthermore, training these models typically requires "paired data"—images of the exact same garment with only one attribute changed—which is virtually non-existent in real-world datasets.

Methodology

TailorGAN addresses these challenges through a specialized architecture and a two-stage training schema that decouples geometry from rendering.

The TailorGAN architecture, illustrating the self-supervised reconstruction and adversarial training paths.

1. Self-Supervised Reconstruction

The model first learns to "repair" a garment. It receives an image where the target attribute (e.g., the collar) is masked out, along with an edge map (a sketch) of the original attribute. By learning to fill this area with the correct original texture, the model establishes a fundamental understanding of the relationship between a garment's overall style and its structural components.

2. Adversarial Learning and Attribute Control

Once the model masters reconstruction, it is tasked with generating a new attribute using an edge map from a different source garment. An attribute-aware discriminator ensures the result is photorealistic and matches the target attribute type (e.g., ensuring a V-neck edge map results in a V-neck image). To maintain the integrity of the rest of the garment, a Self-Attention Mask (SAM) layer identifies exactly which pixels to modify and which to leave untouched.

3. Disentanglement via Edge Maps

Instead of raw RGB images, TailorGAN uses gray-scale edge maps (generated via Holistically-nested Edge Detection) as conditional inputs. This forces the generator to ignore the color and texture of the source attribute, focusing solely on its geometric shape.

Datasets and Evaluation

The researchers introduced GarmentSet, a new benchmark containing 9,636 images with collar annotations and 8,616 with sleeve annotations. Unlike previous datasets, GarmentSet features garments on clean, single-color backgrounds, allowing the model to focus purely on fashion design without background noise.

In comparative tests, TailorGAN significantly outperformed baselines like CycleGAN and Pix2pix:

  • Quantitative Results: In collar translation tasks, TailorGAN achieved a Classification Error (C.E.) of 9.17, compared to 20.01 for Pix2pix, while maintaining higher Structural Similarity (SSIM) scores.
  • Qualitative Results: Visual inspections show that TailorGAN preserves complex textures, such as stripes and logos, across the edit boundary, whereas other models often produced blurry artifacts or "color bulks."

Discussion and Future Work

A key insight of this work is that reconstruction serves as a vital "warm-up" for the latent space; if a model cannot reconstruct an original attribute, it cannot reliably synthesize a new one. By constraining the model with the SAM layer, the authors prevented the "hallucination" of changes in unrelated parts of the garment.

However, limitations remain. The final image quality is heavily dependent on the clarity of the input edge maps. Additionally, complex transitions—such as extending short sleeves to long sleeves—require the model to "invent" significant amounts of new texture, which can occasionally lead to artifacts. Future iterations could see TailorGAN integrated into professional fashion design assistants or e-commerce systems, allowing users to customize designs before purchase.

Summary updated

Abstract

Attribute editing has become an important and emerging topic of computer vision. In this paper, we consider a task: given a reference garment image A and another image B with target attribute (collar/sleeve), generate a photorealistic image which combines the texture from reference A and the new attribute from reference B. The highly convoluted attributes and the lack of paired data are the main challenges to the task. To overcome those limitations, we propose a novel self-supervised model to synthesize garment images with disentangled attributes (e.g., collar and sleeves) without paired data. Our method consists of a reconstruction learning step and an adversarial learning step. The model learns texture and location information through reconstruction learning. And, the model's capability is generalized to achieve single-attribute manipulation by adversarial learning. Meanwhile, we compose a new dataset, named GarmentSet, with annotation of landmarks of collars and sleeves on clean garment images. Extensive experiments on this dataset and real-world samples demonstrate that our method can synthesize much better results than the state-of-the-art methods in both quantitative and qualitative comparisons.
BibTeX
@inproceedings{chen2020tailorgan,
  author = {Lele Chen and Justin Tian and Guo Li and Cheng-Haw Wu and Erh-Kan King and Kuan-Ting Chen and Shao-Hang Hsieh and Chenliang Xu},
  title = {{TailorGAN: Making User-Defined Fashion Designs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=Kek_GNh7ZPyD},
}
Figure