Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation

Pu Cao, Feng Zhou, Lu Yang, Tianrui Huang, Qing SongCVPR 2025 · DOI

Abstract

Large-scale diffusion models like Stable Diffusion excel at open-world generation but often suffer from catastrophic forgetting when fine-tuned on specific image domains. This paper presents a novel guidance-decoupled prior preservation mechanism that enables high-fidelity in-domain generation while maintaining full text-controllability. By separating domain-specific visual knowledge from general control guidance, the proposed method achieves state-of-the-art results using only image datasets, without requiring paired text labels.

Introduction

The advent of large-scale text-to-image diffusion models has revolutionized content creation. However, adapting these models to specific domains (e.g., human faces, specific artistic styles) remains a challenge. Standard fine-tuning often leads to "catastrophic forgetting," where the model gains domain expertise at the cost of its original text-alignment capabilities. For instance, a model fine-tuned on faces might lose the ability to add specific accessories like "glasses" or "hats" if they weren't prevalent in the fine-tuning set.

Overview of the proposed framework and its applications across various domains and tasks.

Existing solutions like LoRA or Textual Inversion often struggle to find the balance between realism and prompt responsiveness. The authors identify that this happens because the internal guidance signals—which dictate domain appearance versus specific attributes—become entangled during standard fine-tuning.

Methods: Guidance-Decoupled Prior Preservation

The authors propose decoupling the internal guidance signals to preserve the pre-trained model's prior knowledge. The core innovation lies in a multi-guidance pipeline that separates "domain guidance" (learned from the new dataset) from "control guidance" (preserved from the original model).

The fine-tuning process highlighting the decoupling of conditional and unconditional guidance to prevent drift and maintain instruction following.

Key components of this approach include:

  1. Conditional Guidance Decoupling: The mathematical guidance signal is split into three components: Domain Guidance (visual distribution), Control Guidance (textual/spatial instructions), and Unconditional Guidance (baseline quality).
  2. Null-text Diffusion Model: A secondary UNet is trained to ignore text, focusing exclusively on the visual features of the target domain. This allows the model to learn domain knowledge efficiently from image-only datasets.
  3. Unconditional Drift Rectification: The researchers identified that even the "unconditional" output (generation without a prompt) drifts during fine-tuning. By using the original model to rectify this drift, they maintain significantly higher photorealism.

Experimental Setup and Results

The method was tested on three diverse datasets: FFHQ (70,000 human faces), AFHQv2 (15,803 animal faces), and a custom Porcelain dataset (5,000 images). These represent both highly structured and textural domains.

The proposed approach significantly outperforms existing baselines like LoRA and Textual Inversion in both quality and controllability:

  • Quantitative Performance: On the FFHQ dataset, the method achieved an FID score of 6.57, nearly doubling the performance of vanilla fine-tuning (12.37) and vastly outperforming LoRA (23.76).
  • Qualitative Success: The model demonstrates a unique ability to combine domain-specific fidelity with complex, open-world prompts, such as generating a porcelain vase with a specific "flamingo pattern" or a face "wearing sunglasses and earrings."

Conclusion and Future Work

By decoupling domain knowledge from control signals, this work enables the creation of specialized diffusion models that remain highly versatile. While the multi-pass inference adds computational overhead compared to single-model approaches, the gain in controllability and fidelity provides a robust path for high-quality, domain-specific image synthesis. Future work will focus on compressing this multi-guidance pipeline into a more efficient single-model architecture.

Summary updated

BibTeX
@inproceedings{cao2025image,
  author = {Pu Cao and Feng Zhou and Lu Yang and Tianrui Huang and Qing Song},
  title = {{Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01711},
}
Figure