HieraFashDiff: Hierarchical Fashion Design with Multi-stage Diffusion Models

Zhifeng Xie, Hao Li, Huiming Ding, Mengtian Li, Xinhan Di, Ying CaoAAAI 2025 · DOI

Introduction

Fashion design is inherently a multi-stage process, transitioning from abstract ideation to granular iteration. Designers typically begin by brainstorming high-level "vibes"—such as "elegant" or "youthful"—to establish a design draft. This is followed by a meticulous refinement phase where specific attributes like sleeve length, collar type, or hemline are adjusted.

Despite the rapid advancement of generative AI, existing models often struggle to replicate this professional workflow. Most current systems are "one-shot" generators that conflate high-level concepts and low-level specifications into a single prompt, resulting in limited control. Furthermore, editing specific features often inadvertently alters the global consistency of the garment. HieraFashDiff addresses these challenges by introducing a hierarchical diffusion-based framework that mirrors the human design process.

The HieraFashDiff Framework

HieraFashDiff decomposes the standard reverse diffusion process into two distinct stages: Ideation and Iteration. By splitting the 1,000 denoising steps, the model allows for a natural progression from coarse global structures to fine-grained details.

Stage 1: Ideation

In the early denoising steps, the model is conditioned on high-level design concepts. This stage focuses on establishing the overall silhouette and "feel" of the garment, resulting in a stable design draft.

Initial design draft generated from high-level concepts like 'Commute' and 'Office'.

Stage 2: Iteration and Attribute Ordering

The later denoising steps are dedicated to refining specific apparel attributes. To maximize editability and maintain structural integrity, the researchers implemented a Spatial Impact Strategy. Attributes are ordered based on the size of the area they affect; for instance, "clothing length" is processed before "collar type." This ensures that large-scale structural changes do not overwrite the fine details added in subsequent steps.

Iterative Local Editing

For precise modifications, HieraFashDiff utilizes a pose-conditioned UNet and the Segment Anything Model (SAM). By incorporating 2D pose maps, the model can accurately localize where specific changes should occur—ensuring a "short sleeve" prompt targets the arms rather than the torso. This allows designers to "blend" new attributes into specific regions while preserving the original fabric texture, color, and model pose.

The full pipeline from design draft generation to sequential, local attribute editing.

The HieraFashion Dataset

A significant contribution of this work is the curation of the HieraFashion dataset. It contains over 5,000 high-resolution (768x1024) images, each meticulously annotated by professional designers. Unlike previous datasets, HieraFashion provides hierarchical descriptions, separating high-level themes from low-level technical attributes, which is essential for training models to understand the design workflow.

Results and Evaluation

HieraFashDiff demonstrates superior performance compared to state-of-the-art models like Stable Diffusion and Attend-and-Excite.

  • Fidelity: The model achieved a Frechet Inception Distance (FID) of 10.27, significantly lower than the 15.33 achieved by the next best model, indicating higher realism.
  • Prompt Adherence: A CLIP-S score of 30.61 confirms that generated images align more closely with complex text prompts.
  • Qualitative Success: Visually, HieraFashDiff is uniquely capable of performing multiple sequential edits—such as changing a dress to short sleeves and then adding a boat neck—without losing the original garment's identity.

Conclusion and Future Work

HieraFashDiff successfully bridges the gap between AI generation and professional fashion design by treating diffusion as a hierarchical process. While currently limited by a relatively small dataset and a reliance on 2D representations, the framework sets a new standard for controllable fashion AI. Future research will look toward expanding dataset diversity and exploring 3D garment generation based on these hierarchical principles.

Summary updated

BibTeX
@inproceedings{xie2025hierafashdiff,
  author = {Zhifeng Xie and Hao Li and Huiming Ding and Mengtian Li and Xinhan Di and Ying Cao},
  title = {{HieraFashDiff: Hierarchical Fashion Design with Multi-stage Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32947},
}
Figure