AIpparel: A Multimodal Foundation Model for Digital Garments

Kiyohiro Nakayama, Jan Ackermann, Timur Levent Kesdogan, Yang Zheng, Maria Korosteleva, Olga Sorkine-Hornung, Leonidas J. Guibas, Guandao Yang, Gordon Wetzstein

TL;DR

AIpparel is the first multimodal foundation model designed to generate and edit digital sewing patterns from text, images, or a combination of both. By introducing a massive new dataset (GCD-MM) and a hyper-efficient tokenization scheme, the researchers have enabled Large Multimodal Models (LMMs) to understand complex garment geometry, achieving a 25x speedup in generation and state-of-the-art accuracy.

What Problem Does This Paper Address?

Creating digital garments is a bottleneck in fashion design, gaming, and the metaverse. The industry standard for these garments is the sewing pattern: a set of flat 2D panels that, when sewn together, form a 3D shape.

Historically, AI models for this task have been "single-modal"—they could handle an image or a text prompt, but rarely both, and they struggled with complex designs. Furthermore, representing the precise geometry of these patterns (vertices, curves, and stitching instructions) in a way that a Large Language Model can understand is incredibly difficult. Previous methods used tens of thousands of tokens to describe a single shirt, making training slow and computationally expensive.

Methods

The authors built AIpparel by fine-tuning LLaVA 1.5-7B using three key innovations:

  1. Efficient Sewing Pattern Tokenization: Instead of treating a pattern as a massive fixed-length vector, they developed a "drawing command" scheme similar to vector graphics. They use special tokens to mark the start/end of panels and specific edge types (lines, Bezier curves, arcs). This reduced the token count per garment by 100x (from ~30k tokens down to ~250).
  2. Hybrid Output Heads: LLMs are great at discrete tokens but poor at precise floating-point coordinates. AIpparel uses the transformer to predict the structure (e.g., "this is a cubic spline"), while small regression heads (MLPs) attached to the hidden states predict the exact vertex positions and 3D transformations.
  3. Positional Embeddings for Geometry: To help the model "see" the geometry it's building, the authors feed vertex positions back into the model as positional embeddings.
Figure 2. Illustration of Our Method. AIpparel uses a novel sewing pattern tokenizer (light blue region) to tokenize each panel into a set of special tokens (light green region). Panel vertex positions and 3D transformations are incorporated using positional embeddings (colored arrows) to the tokens. AIpparel takes in multimodal inputs, such as images and texts (light orange region), to output sewing patterns using autoregressive sampling (light grey region). Finally, the output is decoded to produce simulation-ready sewing patterns (light pink region). See Section 3 for method details.

Datasets

The researchers introduced GCD-MM (GarmentCodeData-MultiModal), the largest dataset of its kind:

  • 120,000 unique garments (10x larger than previous SOTA datasets).
  • Multimodal Annotations: Each entry includes high-quality text descriptions (generated via a hallucination-resistant GPT-4o pipeline), rendered images, and "editing pairs" (before-and-after patterns with natural language instructions like "make the sleeves longer").
  • It significantly increases complexity compared to older datasets like DressCode or SewFactory, featuring diverse edge types and asymmetric designs.

Key Insights and Ideas

  • Structure vs. Precision: The "aha moment" here is the realization that you shouldn't force an LLM to quantize geometry into thousands of tokens. By letting the LLM handle the "grammar" of the garment and specialized heads handle the "math," the model becomes both faster and more accurate.
  • World Knowledge Transfer: Because AIpparel is built on a foundation model, it inherits "common sense." It can reason about abstract prompts like "a dress for a semi-formal garden party," choosing appropriate skirt lengths and styles that weren't explicitly labeled in the training data.
  • Interactive Editing: By training on paired "edit" data, the model unlocks a new workflow where a designer can provide a base image and simply type "add a hood" to update the underlying 2D sewing pattern.

Key Results

  • State-of-the-Art Accuracy: In image-to-garment tasks, AIpparel achieved a Panel L2 error of 5.4 cm, compared to 12.3 cm for the previous best model (SewFormer) on the GCD-MM dataset.
  • Massive Speedup: Due to the efficient tokenization, AIpparel is 25x faster at inference than previous autoregressive models.
  • Simulation-Ready: Unlike baselines that often produce "broken" patterns that fail to drape, AIpparel consistently generates patterns that can be directly used in 3D cloth simulators.
Figure 3. Image-to-Garment Prediction (Qualitative). GCD-MM (Left): our model can reconstruct suitable sewing patterns from the input image alone. In contrast, SewFormer does not produce simulation-ready sewing patterns despite fine-tuning. SewFactory (Right): SewFormer produces inaccurate panels (top row) and incorrect garment types (bottom row) while AIpparel accurately recovers sewing patterns from the images, resulting in superior simulation results. See Sec. 4.1.

Limitations and Future Work

  • Manifold Constraints: The current model is limited to garments representable as 2D surfaces. It cannot yet handle "non-manifold" structures like pockets or complex internal linings.
  • Physicality: The model focuses on geometry but doesn't yet account for physical fabric properties (e.g., how silk drapes differently than denim).
  • Demographic Bias: The authors note that the underlying dataset (GCD) is fitted to statistical models of European and North American adults, which may limit the model's ability to generalize to all body types without further diverse data.

Summary updated

Abstract

Apparel is essential to human life, offering protection, mirroring cultural identities, and showcasing personal style. Yet, the creation of garments remains a time-consuming process, largely due to the manual work involved in designing them. To simplify this process, we introduce AIpparel, a multimodal foundation model for generating and editing sewing patterns. Our model fine-tunes state-of-the-art large multimodal models (LMMs) on a custom-curated large-scale dataset of over 120,000 unique garments, each with multimodal annotations including text, images, and sewing patterns. Additionally, we propose a novel tokenization scheme that concisely encodes these complex sewing patterns so that LLMs can learn to predict them efficiently. AIpparel achieves state-of-the-art performance in single-modal tasks, including text-to-garment and image-to-garment prediction, and enables novel multimodal garment generation applications such as interactive garment editing. The project website is at https://georgenakayama.github.io/AIpparel/.
BibTeX
@inproceedings{nakayama2025aipparel,
  author = {Kiyohiro Nakayama and Jan Ackermann and Timur Levent Kesdogan and Yang Zheng and Maria Korosteleva and Olga Sorkine-Hornung and Leonidas J. Guibas and Guandao Yang and Gordon Wetzstein},
  title = {{AIpparel: A Multimodal Foundation Model for Digital Garments}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00762},
}
Figure