Abstract
Generative models for fashion often struggle with fine-grained semantic alignment, leading to "hallucinations" such as incorrect component counts or misplaced design elements. We present GarmentAligner, a novel text-to-garment diffusion framework that addresses these challenges through a retrieval-augmented contrastive learning strategy and a three-level correction mechanism. By decomposing garments into distinct components and supervising their visual, spatial, and quantitative attributes, GarmentAligner achieves state-of-the-art performance in high-fidelity garment synthesis that strictly adheres to complex design specifications.
Introduction
While general-purpose text-to-image (T2I) models like Stable Diffusion and Midjourney have demonstrated remarkable capabilities, they frequently falter in specialized domains like fashion design. Professional garment synthesis requires a high degree of precision; a prompt for a "denim jacket with two bows on the chest" must result in exactly two bows in the correct anatomical location. Standard models often suffer from three primary deficiencies:
- Quantity Inconsistency: An inability to accurately count specific components (e.g., buttons, pockets).
- Spatial Inconsistency: Misplacement of components relative to the garment's structure.
- Holistic Bias: A tendency to treat garments as monolithic entities rather than structured assemblies of collars, sleeves, and fastenings.
GarmentAligner introduces a component-aware approach to bridge these gaps, ensuring that generated images align with the fine-grained semantics of fashion descriptions.
Methodology
The GarmentAligner framework introduces three major innovations to the diffusion training pipeline to enforce structural and semantic rigor.
1. Automatic Component Extraction
To facilitate component-level supervision, the authors developed an automated pipeline using GroundingDINO for object detection and Segment Anything (SAM) for segmentation. This pipeline processes training images to identify, count, and mask individual garment parts, transforming standard datasets into "component-aware" repositories enriched with spatial and quantitative metadata.
2. Retrieval-augmented Contrastive Learning (RACL)
To sharpen the model's understanding of subtle stylistic differences, GarmentAligner employs a retrieval system during training. For a given prompt, the model retrieves a subset of similar "positive" samples and dissimilar "negative" samples. By applying a contrastive loss, the model learns to distinguish between closely related features—such as the nuance between a "classic collar" and a "v-neck"—which might otherwise be conflated by a standard CLIP-based text encoder.
3. Multi-level Corrections
During the diffusion process, the model is guided by three specific loss functions designed to correct common generative errors:
- Visual Correction: Utilizes CLIP scores to ensure the texture and visual style of a generated component match the textual description.
- Spatial Correction: Aligns the model’s internal cross-attention maps with ground-truth component masks. This forces the model to focus its generative "attention" on the correct pixel regions for specific parts (e.g., ensuring "sleeves" are generated where sleeves should be).
- Quantitative Correction: An integrated detector evaluates the count of generated parts. If the count deviates from the prompt, the model receives a penalty, effectively teaching it to respect numerical constraints.
Experimental Results
Datasets
The model was trained and evaluated on CM-Fashion, a dataset of 500,000 garment images. The authors introduced an Augmented CM-Fashion variant, which includes the component-level segmentation masks and precise quantity counts generated by their extraction pipeline.
Performance Metrics
GarmentAligner was benchmarked against leading models, including SDXL, Midjourney, and specialized architectures like DiffCloth.
- Image Quality and Alignment: GarmentAligner achieved a state-of-the-art FID of 8.735, indicating superior image fidelity, and a CLIPScore of 0.9245, reflecting the highest level of text-image alignment among tested models.
- Structural Accuracy: In targeted evaluations for quantity and spatial accuracy, GarmentAligner outperformed baseline models by 20% to 45%.
- Human Evaluation: In a user study involving 110 participants, GarmentAligner was the preferred model for over 28% of respondents, specifically cited for its superior adherence to complex prompts compared to Midjourney and SDXL.
Discussion and Limitations
GarmentAligner demonstrates that modeling garments as a collection of parts—rather than a single entity—significantly reduces semantic drift. By anchoring words to specific spatial regions through attention-map supervision, the framework provides a blueprint for more controllable fashion AI.
However, the framework's performance is intrinsically linked to the accuracy of the initial GroundingDINO/SAM extraction pipeline; errors in the data-preparation phase can propagate into the generative model. Additionally, as the system is built upon Stable Diffusion v2.1, it inherits certain base-model biases. Future work may involve extending this component-aware logic to 3D garment structures or developing interactive design tools that allow for real-time, text-driven manipulation of specific garment parts.
Summary updated