Abstract
In the realm of fashion design, sketches serve as the canvas for expressing an artist's distinctive drawing style and creative vision, capturing intricate details like stroke variations and texture nuances. The advent of sketch-to-image cross-modal translation technology has notably aided designers. However, existing methods often compromise these sketch details during image generation, resulting in images that deviate from the designer's intended concept. This limitation hampers the ability to offer designers a precise preview of the final output. To overcome this challenge, we introduce HAIFIT, a novel approach that transforms sketches into high-fidelity, lifelike clothing images by integrating multi-scale features and capturing extensive feature map dependencies from diverse perspectives. Through extensive qualitative and quantitative evaluations conducted on our self-collected dataset, our method demonstrates superior performance compared to existing methods in generating photorealistic clothing images. Our method excels in preserving the distinctive style and intricate details essential for fashion design applications. In addition, our method also has obvious advantages in model training and inference speed, contributing to reducing designers' time costs and improving design efficiency.
Introduction
In the world of fashion, a sketch is more than just lines on paper—it’s a blueprint of an artist’s intent, capturing specific stroke weights, textures, and structural nuances. While AI has made massive strides in image generation, most "sketch-to-image" models struggle with the "last mile" of fashion: they often smooth over the very details (like a specific neckline or pocket placement) that make a design unique.
Translating a hand-drawn sketch into a realistic image is a "cross-modal" challenge. Existing methods generally fall into two camps:
- Generative Adversarial Networks (GANs): Fast, but often lose intricate sketch details, resulting in generic-looking clothes.
- Diffusion Models (DMs): High quality, but incredibly slow and computationally expensive, making them impractical for a designer’s iterative workflow.
The core issue is detail retention. Standard convolutional layers treat a sketch as a collection of pixels, often missing the long-range dependencies of a stroke (e.g., how the curve of a sleeve relates to the shoulder). This leads to "hallucinated" details that deviate from the designer's original vision.
Methods
HAIFIT (Human-to-AI Fashion Image Translation) is a GAN-based architecture designed to solve these issues by treating sketches as sequences rather than static grids.
Multi-scale Feature Fusion Encoder (MFFE)
Instead of a standard encoder, HAIFIT uses an Abstract Feature Representation Module (AFRM). This module treats sketch features as a sequence and processes them using a two-layer LSTM. By processing the sketch in both forward and reverse directions, the model captures the "flow" of the lines. This helps the AI understand the designer’s intent and the geometric relationship between different parts of the garment.
Cross-level Skip Connection Module (CSCM)
To ensure that fine details aren't lost as the image is upscaled, the CSCM acts as an attention mechanism. It pulls features from lower-resolution layers and integrates them into higher-resolution ones, ensuring that a pocket or a button detail identified at 64x64 pixels remains sharp and correctly placed at 256x256 pixels.
Pyramid Generative Learning
The model follows a progressive training strategy, starting with low-resolution images and gradually adding layers to reach high-resolution outputs. This allows the model to learn global structure first and fine details later.
Datasets: HAIFashion
The authors found that existing fashion datasets like FashionGEN or DeepFashion were too cluttered with backgrounds and human models to be effective for pure sketch-to-garment translation. To address this, they collected HAIFashion, a dataset of 3,100 background-free fashion images paired with "designer-style" sketches generated via the StyleMe algorithm.
Key Results
HAIFIT was tested against ten state-of-the-art baselines (including pix2pixHD, BBDM, and Palette).
- Structural Fidelity: HAIFIT achieved a Structural Similarity Index (SSIM) of 0.6338, outperforming the second-best model (pix2pixHD) by nearly 13%.
- Perceptual Quality: It led the pack in FID (28.50) and LPIPS (0.077), metrics that measure how "realistic" and "human-aligned" an image looks.
- Speed: Most impressively, HAIFIT is 100x faster in inference than diffusion-based models. This allows for near-instant feedback during the design process.
Conclusion
HAIFIT demonstrates that for specific industrial applications like fashion design, a well-architected GAN can provide superior utility over more complex diffusion models. By treating sketches as sequences and utilizing multi-scale feature fusion, HAIFIT preserves the intricate details essential to a designer's vision. Future work will explore expanding the dataset to more diverse categories and integrating 2D generations into 3D virtual fitting environments.
Summary updated