TL;DR
This paper introduces AIpparel, a large multimodal foundation model that generates and edits digital sewing patterns using text and image inputs. By fine-tuning existing large multimodal models (LMMs) on a novel, large-scale dataset of sewing patterns with rich annotations, AIpparel achieves state-of-the-art performance and enables new applications like interactive garment editing.
What Problem Does This Paper Address?
Creating digital garments is a complex and time-consuming process, largely due to the manual effort involved in designing sewing patterns. Existing methods for generating sewing patterns are typically single-modal, meaning they can only handle one type of input (like text or an image) and struggle to adapt to multimodal inputs or complex garment designs. Furthermore, there's a lack of large-scale, multimodal datasets specifically for sewing patterns, which hinders the development of more sophisticated generative models. This paper aims to bridge this gap by developing a multimodal foundation model capable of understanding and generating sewing patterns from diverse inputs.
Methods
The core of AIpparel is a fine-tuned Large Multimodal Model (LMM), specifically LLaVA-1.5-7B. The authors introduce two key innovations:
- Novel Sewing Pattern Tokenization: To efficiently represent complex sewing patterns for LMMs, they developed a new tokenization scheme. This scheme uses special tokens to denote the start/end of garments and panels, and encodes panel information (name, transformation, edges) concisely. This approach significantly reduces the number of tokens needed compared to previous methods, making fine-tuning more efficient.
- Multimodal Input Handling: AIpparel accepts various inputs, including text descriptions, images, and combinations thereof. It also incorporates a novel fine-tuning objective that allows it to predict both discrete tokens and continuous parameters (like vertex positions and transformations) for the sewing patterns. This is achieved by using small regression heads that map transformer hidden features to these continuous parameters.
Datasets
- GarmentCodeData-MultiModal (GCD-MM): This is a newly curated, large-scale dataset of over 120,000 unique sewing patterns. It expands upon the existing GarmentCodeData (GCD) dataset by adding multimodal annotations, including detailed text descriptions, images, and pairs of original and edited sewing patterns with editing instructions. This dataset is crucial for training and evaluating multimodal garment generation models.
Key Insights and Ideas
- Tokenization is Key for LMMs: The paper highlights that effectively tokenizing complex data like sewing patterns is critical for LMMs to learn and generate them efficiently. Their novel tokenization scheme drastically reduces token count, enabling better performance and faster inference.
- Multimodality Unlocks New Applications: By enabling generation from diverse inputs (text, image, text+image), AIpparel moves beyond single-modal limitations and opens doors to interactive garment editing, where users can modify patterns using natural language.
- Dataset Scale and Richness Matter: The creation of GCD-MM, with its extensive multimodal annotations, is a significant contribution. It provides the necessary foundation for training robust multimodal models for garment generation.
Key Results
- State-of-the-Art Performance: AIpparel achieves state-of-the-art results in image-to-garment and text-to-garment prediction tasks, outperforming existing methods like SewFormer and DressCode by a significant margin on the GCD-MM dataset. For instance, in image-to-garment prediction on GCD-MM, AIpparel achieved a Panel L2 score of 5.4, compared to SewFormer-FT's 12.3.
- Efficiency Gains: The proposed tokenization scheme leads to a substantial speedup, with AIpparel being over 25 times faster at inference compared to DressCode.
- Qualitative Improvements: AIpparel generates more accurate and visually appealing sewing patterns that better align with input descriptions and images. It also demonstrates superior capabilities in language-driven sewing pattern editing, producing more precise modifications.
Limitations and Future Work
- Limitations:
- The current representation is limited to garments representable by manifold surfaces; non-manifold features like pockets require further development.
- The GCD-MM dataset, while large, is based on statistical models of European and North American adults, potentially limiting its representation of diverse body types and styles.
- The model inherits potential biases from the underlying LMMs (like LLaVA) and the annotation generation process (using GPT-4V).
- Future Work:
- Develop efficient representations for non-manifold features.
- Incorporate physical and material constraints for more realistic garment fabrication.
- Expand datasets to include more diverse body types and styles.
- Investigate methods to mitigate biases inherited from foundation models.
- Explore few-shot or in-context generalization for novel garment generation tasks.
Summary updated