Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

Yufan Zhou, Ruiyi Zhang, Jinhui Xu, Tong SunSubmission to ICLR 2024 · Rejected

TL;DR

Current text-to-image personalization techniques rely heavily on regularization during training to prevent models from overfitting, but this often erases fine-grained subject details. ProFusion eliminates training-time regularization entirely by coupling a lightweight encoder network (PromptNet) with a novel two-stage inference procedure (Fusion Sampling). As a result, users can personalize Stable Diffusion to a single reference image in just 5 to 25 seconds while preserving fine details and maintaining flexible text editability.

What Problem Does This Paper Address?

Subject-driven text-to-image customization allows users to place a specific person, pet, or object into new, imaginative scenes specified by text prompts. Most leading methods (such as DreamBooth, Textual Inversion, and E4T) apply regularization techniques during training—like weight decay, explicit $\mathcal{L}_2$ norms on learned token embeddings, or prior-preservation loss—to keep the model from memorizing the reference photo.

However, regularization introduces a steep trade-off:

  • Too little regularization: The learned embedding overpowers the text prompt, preventing creative generation.
  • Too much regularization: The model creates images that match the text, but the unique identity and fine visual details of the subject are washed away.

ProFusion asks: Can we train without regularization to capture exact details, and instead solve the concept-versus-prompt balancing act during inference?

Methods

ProFusion replaces regularization tuning with a synergistic two-part framework:

  1. PromptNet (Concept Encoder):
  2. An encoder initialized from the Stable Diffusion 2 U-Net encoder that takes both the noisy generation latent ($x_t$) and the clean reference image ($\bar{x}_0$) as inputs.
  3. It outputs an unregularized text embedding $S^*$ directly aligned with the SD2 text-conditioning space.
  4. Fine-tuning on a user image is applied only to PromptNet and the cross-attention layers of the diffusion model using a masked diffusion loss and random affine augmentations.
  5. Fusion Sampling (Inference Strategy):
  6. In standard classifier-free guidance, unregularized embeddings easily overpower the prompt text ($C$). Simply splitting guidance into independent condition terms ($S^*$ and $C$) assumes independence, which frequently fails in practice.
  7. ProFusion introduces Fusion Sampling, executing two distinct stages at each denoising step $t$:
  8. Fusion Stage: Uses an update derived from reverse transition distributions (resembling Langevin dynamics) to inject joint information from both $S^*$ and prompt text $C$ into a refined noisy latent $\tilde{x}_t$.
  9. Refinement Stage: Predicts the next step $x_{t-1}$ using decoupled guidance weights for the prompt text and visual embedding, balancing identity preservation against text adherence.

Datasets

The authors evaluate ProFusion using standard personalization and domain datasets:

  • Pre-training: Conceptual Captions (CC3M), using only image inputs for 300k steps to pre-train PromptNet.
  • Exploratory Studies: Flickr-Faces-HQ (FFHQ) to evaluate regularization vs. detail retention.
  • Human Face Evaluation: The benchmark dataset and prompts from E4T (Gal et al., 2023), evaluated against pre-trained face recognition backbones.
  • General Personalization: The DreamBooth benchmark (Ruiz et al., 2022), consisting of 30 diverse subjects across different categories (objects, pets, etc.) tested against 25 distinct prompts per subject.

Key Insights and Ideas

  • Regularization is a blunt instrument: The paper shows that trying to prevent overfitting via embedding norms inherently destroys fine-grained identity features before sampling even begins.
  • Guidance is where the battle should be fought: Overfitting primarily manifests as conditional collapse at inference time (where the text conditioning is ignored in favor of the image embedding). Managing this dynamics during sampling allows models to absorb rich, unregularized embeddings without losing prompt responsiveness.
  • Fast adaptation over deep fine-tuning: Because PromptNet quickly extracts high-capacity embeddings without needing regularization hyperparameter sweeps, effective customization takes only 10 to 50 steps (5 to 25 seconds on a single A100 GPU).

Key Results

  • Human Face Domain: ProFusion achieves higher average image-prompt similarity (across 9 CLIP variants) and higher identity similarity (averaged across ArcFace, AdaFace, FaceNet, DeepFace, etc.) compared to Stable Diffusion, Textual Inversion, DreamBooth, and E4T.
  • DreamBooth Benchmark: Across 30 diverse subjects, ProFusion establishes a superior Pareto frontier in both DINO-based and CLIP-based identity similarity relative to prompt fidelity.
  • Human Evaluation: Amazon Mechanical Turk (MTurk) user studies found that human annotators consistently preferred ProFusion's generated outputs over Stable Diffusion, Textual Inversion, DreamBooth, and E4T.
  • Ablations: Removing the fusion stage caused disjointed compositions (e.g., face structures colliding unnaturally with requested body costumes), while removing the refinement stage degraded fine-grained texture fidelity.

Limitations and Future Work

  • Inference Tuning: The sampling procedure relies on hyperparameter $\gamma$ (scaling factor for the concept embedding in the fusion stage) to balance identity preservation against creativity; higher values favor identity at the expense of prompt compliance.
  • Fine-tuning Requirement: Although adaptation takes as little as 5–25 seconds, it is not completely zero-shot—it still requires fine-tuning PromptNet and cross-attention weights per target subject.
  • Future Directions: The paper does not deeply explore extending this framework to multi-subject composition within a single scene, nor does it provide a formal discussion of long-term failure modes or downstream safety risks (such as deepfake generation).

Summary updated

Abstract

Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concepts specified by a reference image remains a challenging task. To address this problem, researchers have been exploring various methods for customizing pre-trained text-to-image generation models. Currently, most existing methods for customizing pre-trained text-to-image generation models involve the use of regularization techniques to prevent over-fitting. Although regularization will ease the challenge of customization and leads to successful content creation with respect to text guidance, it may restrict the model capability, resulting in the loss of detailed information and inferior performance. In this work, we propose ProFusion, a novel framework for customized text-to-image generation, which can tackle the over-fitting problem without the widely used regularization. Specifically, it consists of an encoder network and a novel sampling method. Given a single user-provided image from an arbitrary domain, the proposed framework can customize a pre-trained text-to-image generation model within half a minute. Empirical results demonstrate that our proposed framework outperforms existing methods.
BibTeX
@misc{zhou2023enhancing,
  author = {Yufan Zhou and Ruiyi Zhang and Jinhui Xu and Tong Sun},
  title = {{Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach}},
  year = {2023},
  eprint = {2305.13579},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13579},
}

Loading related content...

Figure