Abstract
Standard fine-tuning of latent diffusion models often fails to preserve global image properties such as background consistency and subject pose, particularly in domain-specific tasks like retail photography. This paper identifies a fundamental distribution mismatch between the "pure noise" used at inference and the "noised images" seen during training. We propose two techniques—PCA-K Offset Inference and Mean Offset Training—to bridge this gap. By initializing the diffusion process with a noised class-mean or PCA-based image and aligning the training objective accordingly, we demonstrate significant improvements in brand consistency and text-alignment, achieving a 10.6% relative improvement in CLIP scores compared to standard fine-tuning methods.
1. Introduction
In the context of retail photography, consistency is a primary requirement. Professional brand imagery requires uniform white backgrounds, centered products, and specific human poses to maintain a cohesive aesthetic. While one might expect that fine-tuning a model like Stable Diffusion on a large, curated retail dataset would naturally encode these constraints, empirical evidence suggests otherwise. Even after extensive fine-tuning, models often produce erratic backgrounds and inconsistent poses that fail the "vibe check" of a professional catalog.
This paper investigates why standard fine-tuning fails to capture these global properties and proposes a novel initialization strategy to anchor the generative process in the desired distribution.
2. The Distribution Mismatch Problem
The core issue lies in a fundamental assumption of diffusion models: that at the final timestep $T$, a noised image is indistinguishable from pure Gaussian noise $\mathcal{N}(0, I)$. The authors demonstrate that for the image resolutions and noise levels commonly used in practice (such as the $64 \times 64 \times 4$ latent space of Stable Diffusion), this assumption is mathematically flawed.
For a noised image to truly reach the distribution of pure noise, the signal-to-noise ratio must be significantly lower than what is typically achieved at $T$. Consequently, during training, the model "sees" noised versions of structured images (e.g., a centered garment). However, at inference, it is prompted with pure Gaussian noise. This discrepancy leads to out-of-distribution results, as the model struggles to map pure noise back to the specific global structures it was trained on.
3. Proposed Methodology
To bridge the training-inference gap, the authors introduce two complementary methods designed to provide the model with a "global anchor" during the initial stages of denoising.
3.1 PCA-K Offset Inference
Instead of starting the inference process with pure noise, the authors initialize with a heavily noised version of a "base" image. This base image is derived from a Principal Component Analysis (PCA) of the target image class. By utilizing the top $K$ principal components, the model is provided with a "blurry hint" of the expected global structure—such as a centered silhouette and a neutral background—without dictating the fine-grained details which remain controlled by the text prompt.
3.2 Mean Offset Training
Changing the initialization at inference is insufficient on its own, as the model may not know how to interpret the new starting point. The authors modify the training objective so that during the initial denoising steps, the model is trained using the same approximate distribution it will encounter at test time (e.g., the class mean image plus noise). This "Mean Offset Training" ensures that the model learns the relationship between the text prompt and the structured initialization.
4. Experimental Setup and Results
4.1 Dataset
The study utilizes a massive, high-quality retail dataset consisting of over 1 million triplets: a garment image, a human model wearing that garment, and a detailed text description. All images are strictly "on-brand," featuring centered, uncropped subjects on neutral backgrounds with professional lighting.
4.2 Qualitative and Quantitative Performance
Qualitatively, the Mean Offset approach produces images that consistently adhere to professional catalog standards, avoiding the messy backgrounds and awkward cropping seen in standard DDIM fine-tuning.
Quantitatively, the authors employed CLIP Similarity to measure text-alignment. The proposed method improved the CLIP score from 0.2542 (standard) to 0.2812. While the absolute numerical increase is modest, it represents a 10.6% improvement in text-alignment relative to the ground truth baseline. Furthermore, the method integrates seamlessly with ControlNet, providing stabilized backgrounds and lighting in virtual try-on tasks where standard ControlNet often fails.
5. Discussion and Limitations
While effective, the initialization strategy introduces a potential for bias. If the mean image or PCA base contains specific features (e.g., a specific sleeve length or color), the model may struggle to deviate from those "hints," even when prompted to do so. This creates a trade-off between brand consistency and generative variety.
6. Conclusion
By recognizing and addressing the distribution mismatch at timestep $T$, this work provides a practical pathway for deploying diffusion models in high-stakes creative industries like fashion. Future work will explore using Canonical Correlation Analysis (CCA) to dynamically select initializations based on text features, potentially reducing bias and further enhancing the precision of the generated output.
Summary updated