Chat-to-Design: AI Assisted Personalized Fashion Design

Weiming Zhuang, Chongjie Ye, Ying Xu, Pengzhi Mao, Shuai ZhangarXiv 2022 · DOI · arXiv

Abstract

This paper presents Chat-to-Design, a multimodal AI system designed to bridge the gap between fashion search and creation. By integrating natural language processing, CLIP-based retrieval, and StyleGAN-driven editing, the system enables users to co-create personalized apparel. The workflow transitions from a coarse-grained conversational interface to a fine-grained attribute-tuning stage, allowing for real-time adjustments to garment features such as sleeve length and texture. Our results demonstrate that removing background noise from training data significantly improves generative fidelity, providing a seamless end-to-end design experience for non-experts.

Introduction

Traditional e-commerce relies heavily on recommendation systems that track user clicks and ratings to suggest existing products. However, these systems are fundamentally limited by fixed inventory; they cannot satisfy a user's specific vision if the perfect item has not yet been manufactured. While AI has been applied to fashion via chatbots or generative models, these tools are often fragmented. Chat-to-Design addresses this by providing an end-to-end pipeline that allows users to "talk" a garment into existence and then refine its specific attributes in real-time.

Methodology

The Chat-to-Design architecture follows a "coarse-to-fine" workflow, mimicking the relationship between a professional designer and a client. The system is composed of three primary modules:

1. Conversational Interface

The entry point of the system is a natural language understanding (NLU) module built on the Rasa framework and MitieNLP. This module handles intent recognition and entity extraction to establish the user's initial preferences (e.g., "I want an orange dress with short sleeves"). To ensure the bot is "fashion-literate," the authors utilized Chatito to generate synthetic, domain-specific dialog data for training.

The conversational interface allows users to specify initial style preferences through natural language.

2. Multimodal Fashion Retrieval

Once the user's intent is captured, the system employs a retrieval module based on CLIP (Contrastive Language-Image Pre-Training). Fine-tuned on fashion-specific datasets, this module identifies the closest matches in the existing database to serve as a foundation for the design process.

3. Fine-Grained Fashion Design

The final stage involves interactive editing using StyleGAN and StyleFlow. StyleFlow enables attribute-conditioned exploration, allowing users to manipulate specific latent variables of the garment image. This allows for precise control over features like sleeve length, skirt length, and texture density without compromising the overall structural integrity of the design.

Interactive sliders powered by StyleFlow enable precise adjustments to garment attributes like sleeve length and texture.

Data and Training

A critical contribution of this work is the development of a "clean" fashion dataset. While the system utilizes the standard Fashion200K dataset for retrieval, the generative models were trained on a newly constructed dataset of 100,000 images. The authors applied semantic segmentation and edge detection to remove backgrounds and human bodies from these images. This preprocessing ensures that the StyleGAN model focuses exclusively on the garment's structure and texture, avoiding the artifacts and "noise" typically introduced by model poses and busy backgrounds.

Results and Discussion

The system was evaluated primarily on its qualitative performance and interactive capabilities:

  • Visual Fidelity: By training on the "clean" dataset, the system produces garments with sharper details and more realistic textures compared to models trained on raw photography.
  • Interactive Latency: The integration of StyleFlow allows for real-time attribute editing. Users can see the visual impact of their adjustments immediately, which is essential for a collaborative design experience.

Despite these successes, the authors note that the system currently lacks a formal user study to quantify the "design-capability" of non-expert users. Additionally, future iterations could benefit from more rigorous quantitative benchmarks such as FID scores.

Conclusion

Chat-to-Design demonstrates the power of integrating NLU, computer vision, and generative AI into a single, cohesive workflow. By moving from coarse conversation to fine-grained editing, the system empowers users to become active participants in the design process. Future work will explore the integration of virtual try-on technology and the potential for generating personalized digital apparel for the metaverse.

Summary updated

Abstract

In this demo, we present Chat-to-Design, a new multimodal interaction system for personalized fashion design. Compared to classic systems that recommend apparel based on keywords, Chat-to-Design enables users to design clothes in two steps: 1) coarse-grained selection via conversation and 2) fine-grained editing via an interactive interface. It encompasses three sub-systems to deliver an immersive user experience: A conversation system empowered by natural language understanding to accept users' requests and manages dialogs; A multimodal fashion retrieval system empowered by a large-scale pretrained language-image network to retrieve requested apparel; A fashion design system empowered by emerging generative techniques to edit attributes of retrieved clothes.
BibTeX
@misc{zhuang2022chat,
  author = {Weiming Zhuang and Chongjie Ye and Ying Xu and Pengzhi Mao and Shuai Zhang},
  title = {{Chat-to-Design: AI Assisted Personalized Fashion Design}},
  year = {2022},
  eprint = {2207.01058},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.01058},
}
Figure