Abstract
GarmentDiffusion is a novel multimodal generative model designed to bridge the gap between creative fashion concepts and physical production. By generating precise, vectorized 3D sewing patterns from text, sketches, or partial designs, it addresses the limitations of previous unimodal and computationally expensive autoregressive models. Utilizing a Diffusion Transformer (DiT) architecture combined with a compact edge-encoding scheme, GarmentDiffusion achieves a 100x speedup in generation while maintaining centimeter-level accuracy and scalability to complex garment designs.
Introduction
In the fashion industry, the bridge between a creative concept, such as a sketch or a text description, and a physical product is the sewing pattern. These patterns consist of a set of 2D panels that, when stitched together, form a 3D garment. Digitally generating these patterns is a significant challenge because they are geometrically complex, involving a variable number of panels and edges with precise curvature and connectivity requirements.
Existing AI approaches for pattern generation suffer from two primary bottlenecks. First, most models are unimodal, restricting designers to a single input type (e.g., only images). Second, previous state-of-the-art models like SewingGPT rely on autoregressive architectures that predict parameters sequentially. For complex garments, this results in massive sequence lengths—up to 18,000 tokens—making training and inference prohibitively slow and prone to error accumulation.
Methods
GarmentDiffusion introduces a paradigm shift from autoregressive generation to parallel denoising using a Diffusion Transformer (DiT). The core of this approach lies in two technical innovations: a novel data representation and a flexible conditioning mechanism.
Edge-Oriented Representation
To solve the sequence length problem, the researchers developed an edge-oriented encoding scheme. Instead of treating every coordinate as an individual token, the model packs all parameters for a single edge—including endpoints, Bézier control points, arc parameters, and stitch tags—into the embedding dimension of a single token. This compact representation reduces the total sequence length by a factor of ten, allowing the model to handle highly intricate designs without exceeding memory limits.
Diffusion Transformer (DiT) Architecture
The model utilizes a DiT architecture to denoise these edge tokens simultaneously. Unlike sequential models, GarmentDiffusion processes the entire garment structure in parallel across a fixed number of denoising steps. This parallelization is the primary driver behind the model's dramatic speed improvements.
Multimodal Conditioning
To support diverse creative workflows, the model incorporates frozen CLIP encoders for both text and images. These features are integrated into the DiT blocks via decoupled cross-attention layers. This design allows the model to weigh text and visual prompts independently, enabling high-fidelity generation from brief descriptions, detailed technical specifications, or visual sketches. Furthermore, by replacing a portion of the initial Gaussian noise with known pattern data, the model can perform "pattern completion," assisting designers in finishing partial designs.
Datasets and Evaluation
The model was rigorously evaluated on three major benchmarks: SewFactory (~13,700 patterns), DressCodeData (~19,700 patterns), and GarmentCodeData (~115,000 patterns). To improve the quality of text-based generation, the authors used Large Language Models (LLMs) to generate multi-level text descriptions for the DressCodeData set, ranging from brief category labels to detailed descriptions of component geometry (e.g., collar depth and sleeve curvature).
Key Results
GarmentDiffusion demonstrates significant improvements over existing benchmarks in both efficiency and quality:
- Efficiency: The model achieves a 100x speedup compared to SewingGPT. It can generate a complete, production-ready 3D sewing pattern in under 0.5 seconds on a single NVIDIA A10 GPU.
- Accuracy: On the DressCodeData benchmark, GarmentDiffusion outperformed previous models across all geometric metrics, specifically achieving a lower Panel L2 distance, which indicates higher fidelity to ground-truth designs.
- Scalability: It is the first model capable of processing the full complexity of the GarmentCodeData set, which contains designs too large for previous autoregressive transformers to handle.
- Production Readiness: Qualitative results show that the generated patterns maintain realistic physical properties when draped onto 3D human models, making them suitable for professional fashion design pipelines.
Conclusion and Future Work
GarmentDiffusion represents a major step forward in automated garment design, proving that diffusion-based parallel denoising is superior to autoregressive modeling for geometric tasks like sewing pattern generation. While the model excels at capturing complex geometry, future work will focus on improving explicit stitching connectivity and integrating numerical controls to allow designers to specify exact measurements. Additionally, conditioning generation on specific human body shapes could further enhance the utility of the model for custom tailoring applications.
Summary updated