Tailor Me: An Editing Network for Fashion Attribute Shape Manipulation

Youngjoong Kwon, Stefano Petrangeli, Dahun Kim, Haoliang Wang, Viswanathan Swaminathan, Henry FuchsWACV 2022 · OpenReview

Abstract

Fashion attribute editing aims to manipulate fashion images based on a user-specified attribute, while preserving the details of the original image as intact as possible. Recent works in this domain have mainly focused on direct manipulation of the raw RGB pixels, which only allows to perform edits involving relatively small shape changes (e.g., sleeves). The goal of our Virtual Personal Tailoring Network (VPTNet) is to extend the editing capabilities to much larger shape changes of fashion items, such as cloth length. To achieve this goal, we decouple the fashion attribute editing task into two conditional stages: shape-then-appearance editing. To this aim, we propose a shape editing network that employs a semantic parsing of the fashion image as an interface for manipulation. Compared to operating on the raw RGB image, our parsing map editing enables performing more complex shape editing operations. Second, we introduce an appearance completion network that takes the previous stage results and completes the shape difference regions to produce the final RGB image. Qualitative and quantitative experiments on the DeepFashion-Synthesis dataset confirm that VPTNet outperforms state-of-the-art methods for both small and large shape attribute editing.

Introduction

Traditional fashion editing networks, such as StarGAN or AttGAN, typically attempt to manipulate raw RGB pixels directly. While effective for color changes or minor adjustments, this approach struggles with significant geometric transformations. Direct RGB manipulation faces two primary challenges: geometric complexity and identity preservation. Large-scale changes, like altering garment length, often result in blurry artifacts or "ghosting" as the network struggles to maintain human anatomy. Furthermore, because these models often regenerate the entire image from scratch, they frequently alter non-target regions such as the wearer's face or the background.

VPTNet (Virtual Personal Tailoring Network) addresses these issues by introducing a "shape-then-appearance" decoupling strategy. By first manipulating a semantic parsing map—a structural blueprint—and then inpainting the resulting image, the system enables dramatic shape changes while preserving the wearer’s identity and fine-grained textures.

Methodology

The VPTNet architecture consists of a two-stage pipeline that separates structural manipulation from visual synthesis.

Stage 1: Shape Editing Network ($G^P$)

Instead of editing pixels, this stage operates on a semantic parsing map where different regions (e.g., hair, face, arms, upper-cloth) are represented by distinct labels.

  • Target Region Localization (TRL): This module utilizes channel-wise and spatial attention to identify specific regions of the parsing map that require modification based on the user's attribute request (e.g., "make sleeves longer").
  • Edge-Preserving Constraints: To ensure structural integrity and prevent "blobby" outputs, the network simultaneously predicts an edge map. This acts as a regularizer, keeping the boundaries between the skin and clothing crisp and realistic.

Stage 2: Appearance Completion Network ($G^I$)

Following the generation of the edited parsing map, the system calculates a difference mask ($M_{diff}$) between the original and edited shapes.

  • Selective Inpainting: The network performs inpainting only within the areas defined by the difference mask. By retaining original pixels for the face, hair, and background, the model ensures 100% identity preservation in non-edited regions.
  • Gated Convolutions: The inpainter employs gated convolutions to effectively handle the irregular shapes of the newly generated garment regions, ensuring seamless integration with the original texture.

Experimental Results

The model was evaluated using the DeepFashion-Synthesis dataset, comprising approximately 79,000 images. Since the dataset lacked specific labels for cloth dimensions, the authors generated pseudo-labels based on area ratios within the parsing maps.

Figure 4: Examples of sleeve length manipulation using VPTNet, demonstrating the model's ability to handle both additions and removals of garment sections.

Quantitative Performance

VPTNet significantly outperformed state-of-the-art models like AMGAN and Fashion-AttGAN across several metrics:

  • Attribute Accuracy: In classification tests to verify if edits matched requested attributes, VPTNet achieved 85.81% accuracy, surpassing the next best model (~81.8%).
  • Image Fidelity: On the Structural Similarity Index (SSIM), VPTNet scored 0.9862, indicating that edited images remain remarkably close to the original's visual quality.
  • User Preference: A study with 63 participants showed a strong preference for VPTNet, with 69.8% favoring its attribute generation over competitors.

Discussion and Conclusion

A key insight of this work is that semantic parsing maps provide a stable latent space for shape manipulation, as they are less noisy and follow stricter structural rules than raw pixels. By explicitly calculating a difference mask, the model avoids the "hallucination" problem common in GANs, where unintended changes occur in non-target regions.

While VPTNet excels at shape manipulation, its performance is currently dependent on the accuracy of the initial parsing map estimator. Future work may explore integrating a dedicated appearance editor to handle complex texture transfers and scaling the architecture to higher resolutions for professional fashion photography applications.

Summary updated

Abstract

Fashion attribute editing aims to manipulate fashion images based on a user-specified attribute, while preserving the details of the original image as intact as possible. Recent works in this domain have mainly focused on direct manipulation of the raw RGB pixels, which only allows to perform edits involving relatively small shape changes (e.g., sleeves). The goal of our Virtual Personal Tailoring Network (VPTNet) is to extend the editing capabilities to much larger shape changes of fashion items, such as cloth length. To achieve this goal, we decouple the fashion attribute editing task into two conditional stages: shape-then-appearance editing. To this aim, we propose a shape editing network that employs a semantic parsing of the fashion image as an interface for manipulation. Compared to operating on the raw RGB image, our parsing map editing enables performing more complex shape editing operations. Second, we introduce an appearance completion network that takes the previous stage results and completes the shape difference regions to produce the final RGB image. Qualitative and quantitative experiments on the DeepFashion-Synthesis dataset confirm that VPTNet outperforms state-of-the-art methods for both small and large shape attribute editing.
BibTeX
@inproceedings{kwon2022tailor,
  author = {Youngjoong Kwon and Stefano Petrangeli and Dahun Kim and Haoliang Wang and Viswanathan Swaminathan and Henry Fuchs},
  title = {{Tailor Me: An Editing Network for Fashion Attribute Shape Manipulation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=Zfz7_-WuXcU},
}
Figure