Abstract
Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concept provided by the user input image is still a challenging task. To address this problem, researchers have been exploring various methods for customizing pre-trained text-to-image generation models. Currently, most existing methods for customizing pre-trained text-to-image generation models involve the use of regularization techniques to prevent over-fitting. While regularization will ease the challenge of customization and leads to successful content creation with respect to text guidance, it may restrict the model capability, resulting in the loss of detailed information and inferior performance. In this work, we propose a novel framework for customized text-to-image generation without the use of regularization. Specifically, our proposed framework consists of an encoder network and a novel sampling method which can tackle the over-fitting problem without the use of regularization. With the proposed framework, we are able to customize a large-scale text-to-image generation model within half a minute on single GPU, with only one image provided by the user. We demonstrate in experiments that our proposed framework outperforms existing methods, and preserves more fine-grained details.
Introduction
Large-scale text-to-image (T2I) models like Stable Diffusion have revolutionized content creation, yet they often struggle to accurately represent "novel concepts"—specific subjects provided by a user, such as a particular person's face or a unique object. Existing customization techniques, such as DreamBooth or Textual Inversion, typically rely on regularization to prevent the model from overfitting to the limited input data. However, this regularization often acts as a "blur," washing out the unique, fine-grained details that define the subject's identity.
This paper introduces ProFusion, a novel framework that eliminates the need for training-time regularization. By shifting the burden of preventing overfitting to the inference phase, ProFusion maintains high identity preservation while ensuring the model remains responsive to creative text prompts.
Methodology
The ProFusion framework is built upon three primary components designed to capture and preserve intricate details without the constraints of traditional regularization.
PromptNet Encoder
At the heart of the system is PromptNet, a 5-block encoder network. It maps a single input image into a specific word embedding ($S^*$) within the Stable Diffusion text-embedding space. Crucially, PromptNet is trained without $L_2$ regularization on the embedding, allowing it to capture a much richer set of subject-specific features than previous methods.
Fusion Sampling
To manage the potential for overfitting during generation, the authors introduce Fusion Sampling. Unlike standard Classifier-Free Guidance (CFG), which can be overwhelmed by strong subject embeddings, Fusion Sampling employs a two-stage process at each denoising step:
- Fusion Stage: Treats the condition as a gradient-based optimization, injecting subject ($S^*$) and text ($C$) information into the noisy latent.
- Refinement Stage: Uses a modified sampling rule derived from Bayes' Rule to balance the subject and text conditions independently, preventing the subject's identity from overpowering the requested style.
Efficient Fine-Tuning
The framework utilizes data augmentation via an inpainting model to generate diverse views of the subject. This allows the model to decouple the subject from its original background during a rapid 50-step fine-tuning phase, which takes approximately 30 seconds on a single GPU.
Experimental Evaluation
The authors pre-trained the PromptNet encoder on 80,000 iterations of the FFHQ (Flickr-Faces-HQ) dataset. For evaluation, they compared ProFusion against state-of-the-art baselines, including DreamBooth, Textual Inversion, and E4T, using a set of 39 source image/prompt pairs. Identity preservation was quantitatively measured using cosine similarity from pre-trained face recognition models like ArcFace and DeepFace.
Results
ProFusion demonstrates a significant advancement in both identity preservation and text adherence.
Quantitative and Qualitative Performance
Quantitatively, ProFusion achieved a DeepFace similarity score of 0.774, notably higher than E4T (0.732) and DreamBooth (0.643). Qualitatively, the model excels at maintaining recognizable facial features across diverse styles, as shown in Figure 1.
Human Evaluation
In subjective tests conducted via Amazon Mechanical Turk, users consistently preferred ProFusion's outputs. The model's ability to follow complex prompts like "in Ghibli anime style" while retaining the subject's likeness resulted in preference margins often exceeding 70% against competing methods.
Conclusion
ProFusion offers a robust, regularization-free approach to text-to-image customization. By leveraging a specialized encoder and a novel inference-time sampling strategy, it achieves superior detail preservation and editability with minimal computational overhead. While Fusion Sampling increases inference time compared to standard methods, the gains in fidelity and speed of customization make it a compelling solution for personalized image generation.
Summary updated