Introduction
Text-to-image (T2I) customization aims to teach pre-trained models like Stable Diffusion to recognize and generate specific visual concepts—such as a personal pet or a unique object—from a few reference images. However, existing methods like DreamBooth or LoRA frequently encounter two forms of overfitting. Concept-agnostic overfitting causes the model to "forget" general knowledge, leading to unrelated generations resembling the target concept. Concept-specific overfitting results in a "copy-paste" effect, where the model becomes locked to the backgrounds, poses, and layouts of the training data, losing its creative flexibility.
Methodology: Decoupling "Where" from "What"
The authors introduce Infusion, a lightweight customization framework (requiring only 11KB of storage) that addresses these challenges by decoupling the spatial and semantic components of the cross-attention module.
Infusion operates on the insight that foundational models already possess the "imagination" to compose scenes; they simply lack the specific identity of the new concept. The architecture utilizes a Dual-Stream Pipeline:
- Foundational Stream: Uses the original model to generate attention maps, which dictate the "where" (geometry and layout).
- Customized Stream: Hijacks these maps and applies a Residual Value Embedding ($\Delta$). This learned residual is added only to the value features of the concept token (e.g., "corgi"), injecting the "what" (visual identity) without altering the model's structural knowledge.
This approach treats overfitting as a distribution collapse in the latent space. By restricting updates to a tiny residual, Infusion preserves the wide distribution of modalities found in the base model.
Evaluation and Metrics
To rigorously measure overfitting, the researchers introduced two new metrics:
- Latent Fisher Divergence: Measures the degree to which the model forgets general knowledge by comparing its performance on a subset of the LAION-400M dataset before and after customization.
- Wasserstein Metric: Quantifies the loss of diversity within the customized concept's generations.
Key Results
Infusion demonstrates superior performance in balancing fidelity with text alignment. In evaluations across 20 distinct subjects, the method achieved a CLIP-T score of 0.267, outperforming established baselines like DreamBooth (0.255) and Perfusion (0.242).
Furthermore, Infusion is remarkably robust to training duration. Unlike other methods that collapse if trained for too many iterations, Infusion's metrics remained stable up to 2,000 steps. In a user study, participants preferred Infusion over 50% of the time when compared head-to-head with Textual Inversion, Custom Diffusion, and DreamBooth for both image fidelity and prompt adherence.
Conclusion
Infusion offers a highly efficient and effective solution for T2I customization. By isolating identity updates to a minimal residual embedding and leveraging the foundational model's inherent layout capabilities, it prevents the catastrophic forgetting and diversity loss typical of larger fine-tuning approaches. Its 11KB footprint makes it an ideal candidate for scalable, multi-concept applications in creative workflows.
Summary updated