Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation

Gayatri Deshmukh, Somsubhra De, Chirag Sehgal, Jishu Sen Gupta, Sparsh MittalarXiv 2024 · DOI · arXiv

Abstract

This paper introduces FLORA, the first comprehensive dataset of 4,330 fashion sketches paired with professional-grade textual descriptions, and NeRA, a novel adapter architecture. By replacing the standard linear transformations in Parameter-Efficient Fine-Tuning (PEFT) with non-linear Kolmogorov-Arnold Networks (KANs), the authors achieve superior semantic alignment and faster convergence in generating high-fidelity fashion outfits from text.

Introduction: The "Last Mile" of Fashion AI

While AI has made massive strides in general image generation, the fashion industry presents a unique "last mile" challenge. Professional fashion design relies on sketches—abstract, stylistically rich representations—rather than photorealistic images. Existing models struggle here for two primary reasons:

  1. Data Scarcity: There are few datasets that pair fashion sketches with the specific, technical jargon used by designers (e.g., "asymmetrical neckline," "tiered ruffles," or "double-breasted waistcoat").
  2. Architectural Rigidity: Popular fine-tuning methods like LoRA (Low-Rank Adaptation) rely on linear transformations. However, the relationship between a complex textual description and a nuanced visual style is inherently non-linear. Standard adapters often fail to capture the subtle stylistic elements required for high-end design.

The FLORA Dataset Pipeline

To address the data gap, the authors developed FLORA (Fashion Language Outfit Representation for Apparel Generation). This dataset consists of 4,330 curated pairs of fashion sketches and technical descriptions. The creation process involved a rigorous multi-stage pipeline to ensure high quality and professional relevance.

Figure 1: The FLORA dataset includes diverse garment types and a multi-stage cleaning pipeline for internet-scraped images.

The pipeline includes several critical steps:

  • VLM Filtering: Using LLaVa to ensure images contained only a single outfit sketch, filtering out noise from the initial scrape of 10,000 images.
  • Automated Cleanup: Employing Keras-OCR and ControlNet to remove watermarks and signatures without damaging the underlying sketches.
  • Expert Annotation: Leveraging GPT-4o to generate descriptions structured around model pose, outfit details, color, and accessories, specifically using industry-standard terminology.
Figure 3: An example from the FLORA dataset showing a fashion sketch paired with a detailed, technical description of the model, outfit, and accessories.

The NeRA Adapter Architecture

The NeRA (Nonlinear low-rank Expressive Representation Adapter) is the paper's technical centerpiece. Most adapters (like LoRA or DoRA) use Multi-Layer Perceptrons (MLPs) with fixed activation functions. NeRA instead utilizes Kolmogorov-Arnold Networks (KANs).

Unlike MLPs, which have weights on edges, KANs have learnable, non-linear activation functions (splines or Radial Basis Functions) on the edges. This allows NeRA to model complex, high-order relationships between text and pixels more efficiently than linear methods. It acts as a "drop-in" replacement for MLP layers in large diffusion models like FLUX.1 or Stable Diffusion 3.

Key Insights and Results

The research highlights that non-linearity is essential for creativity. Fashion design isn't a linear mapping; for instance, a "mermaid gown" involves specific geometric constraints that linear adapters struggle to maintain when other variables change. NeRA’s spline-based transformations handle these dependencies much better.

Performance Benchmarks

The authors tested NeRA against state-of-the-art adapters (LoRA, LoKR, DoRA, LoHA) across multiple base models:

  • Quantitative Superiority: On the FLUX model, NeRA achieved an FID (Fréchet Inception Distance) of 6.05, significantly beating LoRA’s 7.88. Lower FID indicates higher image realism and diversity.
  • Semantic Alignment: NeRA scored higher on CLIP-SIM (0.3412 vs LoRA’s 0.2987), proving that the generated images matched the complex prompts more accurately.
  • Efficiency: Despite its complexity, NeRA reached optimal performance in significantly fewer training steps than LoRA, reducing total training time by nearly half in some configurations.

In a blind human evaluation with 500 participants, NeRA-generated sketches were rated higher in Creativity, Prompt Relevance, and Visual Quality compared to all other PEFT methods.

Conclusion and Future Directions

While a major step forward, the authors identify several areas for future work. Currently, the model generates the whole outfit; future iterations aim to allow "part-level control" for editing specific elements like sleeves or necklines. Additionally, the team plans to expand the FLORA dataset to include more diverse cultural styles and optimize KAN-based inference for real-time mobile applications.

Summary updated

Abstract

Specialized datasets that capture the fashion industry's rich language and styling elements can boost progress in AI-driven fashion design. We present FLORA, (Fashion Language Outfit Representation for Apparel Generation), the first comprehensive dataset containing 4,330 curated pairs of fashion outfits and corresponding textual descriptions. Each description utilizes industry-specific terminology and jargon commonly used by professional fashion designers, providing precise and detailed insights into the outfits. Hence, the dataset captures the delicate features and subtle stylistic elements necessary to create high-fidelity fashion designs. We demonstrate that fine-tuning generative models on the FLORA dataset significantly enhances their capability to generate accurate and stylistically rich images from textual descriptions of fashion sketches. FLORA will catalyze the creation of advanced AI models capable of comprehending and producing subtle, stylistically rich fashion designs. It will also help fashion designers and end-users to bring their ideas to life. As a second orthogonal contribution, we introduce NeRA (Nonlinear low-rank Expressive Representation Adapter), a novel adapter architecture based on Kolmogorov-Arnold Networks (KAN). Unlike traditional PEFT techniques such as LoRA, LoKR, DoRA, and LoHA that use MLP adapters, NeRA uses learnable spline-based nonlinear transformations, enabling superior modeling of complex semantic relationships, achieving strong fidelity, faster convergence and semantic alignment. Extensive experiments on our proposed FLORA and LAION-5B datasets validate the superiority of NeRA over existing adapters. We will open-source both the FLORA dataset and our implementation code.
BibTeX
@misc{deshmukh2024dressing,
  author = {Gayatri Deshmukh and Somsubhra De and Chirag Sehgal and Jishu Sen Gupta and Sparsh Mittal},
  title = {{Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation}},
  year = {2024},
  eprint = {2411.13901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13901},
}
Figure