From Design Draft to Real Attire: Unaligned Fashion Image Translation

Yu Han, Shuai Yang, Wenjing Wang, Jiaying Liu

Abstract: Fashion design often begins with stylized sketches that utilize exaggerated proportions and poses, creating a significant structural gap between the initial draft and the final product. Traditional image-to-image translation models struggle with this "alignment gap," often producing artifacts or failing to preserve intricate textures. This paper introduces D2RNet, a novel framework designed for unaligned fashion image translation. By employing a two-stream architecture—comprising a detail preservation stream and a shape correction stream—alongside a saliency-based sampling layer, D2RNet effectively warps design drafts to realistic proportions while maintaining fine-grained patterns. Experimental results demonstrate that D2RNet outperforms existing state-of-the-art models in both quantitative metrics and qualitative human evaluation.

Introduction

In the fashion industry, the journey from concept to creation starts with a hand-drawn sketch. These drafts are intentionally stylized, featuring "fashion proportions" such as elongated limbs and dramatic poses that do not correspond to real-world human geometry. Translating these sketches into photorealistic images is a critical task for the fashion industry, yet it presents a unique challenge: unaligned data.

A typical stylized fashion design draft featuring exaggerated proportions.

Standard translation models like Pix2pix or SPADE assume pixel-wise alignment, leading to distortions when faced with the structural discrepancies inherent in fashion drafts. Furthermore, standard computer vision tools often fail on smooth, featureless design drafts, making it difficult to use skeletal landmarks to guide the alignment.

Methodology: The D2RNet Framework

To address the alignment gap, researchers from Peking University developed D2RNet (Design to Real Network). The core of the framework is a saliency-based sampling layer that adaptively warps the input draft to roughly align with the target real item's structure. This is followed by a two-stream network architecture:

  1. Detail Preservation Stream: This stream focuses on capturing and maintaining the intricate textures, prints, and patterns of the original draft.
  2. Shape Correction Stream: This stream focuses on adjusting the garment's boundaries and proportions to match realistic human forms.

A fusion network then uses a learned mask to intelligently combine the best parts of both streams into one crisp image. Additionally, the authors presented R2DNet (Real to Design), a reverse model that can translate a photo of a real person onto a design-style model, effectively acting as a stylized virtual try-on tool.

Results and Evaluation

D2RNet was evaluated on a newly curated Fashion Design Draft Dataset containing 1,173 unaligned pairs of design drafts and their corresponding real-world fashion items. In comparative studies against heavyweights like CycleGAN, Pix2pixHD, and SPADE, D2RNet achieved superior performance.

A photorealistic translation of a fashion item, demonstrating the goal of the D2RNet framework.

In user studies, D2RNet received a mean score of 4.94, significantly higher than Pix2pix (3.29) and Pix2pixHD (3.50). It also achieved the highest SSIM (Structural Similarity Index) score of 0.645. Qualitatively, D2RNet was uniquely able to preserve fine-grained patterns (like small text or complex lace) while simultaneously correcting the "long-leg" distortion of the original sketches. Other models either blurred the textures or failed to adjust the garment's shape to a realistic human proportion.

Conclusion

D2RNet provides a robust solution for the unaligned image translation problem in fashion design. By prioritizing warping and detail preservation over pure synthesis, the model bridges the gap between artistic vision and realistic representation. The authors suggest this "unaligned translation" approach could be applied to other fields with structural gaps, such as photo-to-caricature or photo-to-emoji translation, where the source and target have fundamentally different geometries.

Summary updated

Abstract

Fashion manipulation has attracted growing interest due to its great application value, which inspires many researches towards fashion images. However, little attention has been paid to fashion design draft. In this paper, we study a new unaligned translation problem between design drafts and real fashion items, whose main challenge lies in the huge misalignment between the two modalities. We first collect paired design drafts and real fashion item images without pixel-wise alignment. To solve the misalignment problem, our main idea is to train a sampling network to adaptively adjust the input to an intermediate state with structure alignment to the output. Moreover, built upon the sampling network, we present design draft to real fashion item translation network (D2RNet), where two separate translation streams that focus on texture and shape, respectively, are combined tactfully to get both benefits. D2RNet is able to generate realistic garments with both texture and shape consistency to their design drafts. We show that this idea can be effectively applied to the reverse translation problem and present R2DNet accordingly. Extensive experiments on unaligned fashion design translation demonstrate the superiority of our method over state-of-the-art methods. Our project website is available at: https://victoriahy.github.io/MM2020/ .
BibTeX
@inproceedings{han2020from,
  author = {Yu Han and Shuai Yang and Wenjing Wang and Jiaying Liu},
  title = {{From Design Draft to Real Attire: Unaligned Fashion Image Translation}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413953},
}
Figure