Abstract
Personalizing text-to-image diffusion models to recognize specific user-defined subjects is a significant challenge when multiple concepts are introduced simultaneously. Existing methods often suffer from overfitting to training layouts or "attribute leakage," where features of different subjects blend together. This paper introduces FaR (Fuse-and-Refine), a fine-tuning framework designed to mitigate these issues. By combining a Concept Fusion data augmentation strategy with a Localized Refinement loss function, FaR enables the generation of multiple distinct subjects in diverse compositions while maintaining high identity fidelity and text alignment.
Introduction
Recent advancements in diffusion models have made it possible to personalize image generation using only a few reference images of a specific subject (e.g., a user's pet or a unique object). While single-subject personalization is relatively mature, multi-concept personalization—generating a scene containing multiple specific subjects—remains a significant hurdle.
Two primary problems plague current multi-concept models:
- Overfitting: Due to the extremely small size of training datasets (typically 3–5 images), models often memorize the exact spatial configuration of the training photos, limiting their ability to generate subjects in new poses or environments.
- Attribute Leakage: When multiple subjects are present, especially those of similar categories (e.g., two different dogs), the model frequently mixes their visual traits. This results in hybrid subjects that lack the distinct identity of either reference.
To address these challenges, we propose the FaR framework, which focuses on improving the model's understanding of subject boundaries and compositional diversity during the fine-tuning process.
The FaR Framework
1. Concept Fusion: Diversifying the Training Set
The "Fuse" component of FaR addresses overfitting through synthetic data augmentation. Rather than relying solely on the original static images, the framework utilizes Grounded SAM to segment the target subjects. These segmented subjects are then randomly "pasted" onto diverse backgrounds at varying scales and positions.
To ensure the model retains general category knowledge, the framework also incorporates "prior images"—generic versions of the subject's class (e.g., a generic "chow chow") generated by a base Stable Diffusion model. This synthetic dataset teaches the model how specific subjects should look in a wide variety of spatial arrangements, breaking the reliance on the original training layouts.
2. Localized Refinement: Preventing Attribute Leakage
The "Refine" component introduces a novel loss function based on the model's cross-attention maps. These maps represent the spatial regions the model focuses on when processing specific text tokens.
FaR enforces a spatial constraint: the attention map for a specific concept’s token must align strictly with the actual segmentation mask of that object. By mathematically penalizing the model when the "attention" for one subject overlaps with the pixels of another, the framework forces the model to isolate identities within their respective physical boundaries. This prevents the visual attributes of Subject A from bleeding into Subject B.
3. Targeted Fine-tuning Strategy
Rather than updating the entire weights of the diffusion model, which can lead to catastrophic forgetting of general concepts, FaR targets three specific components: the text encoder, the self-attention layers, and the cross-attention layers. This selective approach balances the acquisition of new, specific details with the preservation of the model's original generative capabilities.
Experimental Evaluation
Datasets and Metrics
The researchers evaluated FaR using a comprehensive dataset of 24 distinct concepts, including humans, animals, and objects, sourced from DreamBench, CustomConcept101, and Mix-of-show. The evaluation involved 1,200 generated images across 16 multi-concept combinations.
Performance was measured using two primary metrics:
- D&C-DS (Image Alignment): Measures how accurately the generated image matches the reference subjects.
- ImageReward (Text Fidelity): Evaluates how well the generated image follows the user's prompt (e.g., "in the snow").
Results
FaR demonstrated superior performance compared to state-of-the-art baselines like Custom Diffusion, Textual Inversion, and ConesV2.
- Quantitative Performance: FaR achieved a D&C-DS score of 0.849 for single-subjects and 0.664 for multi-subjects, significantly outperforming Mix-of-Show (0.791 and 0.480, respectively).
- Qualitative Results: Visual analysis confirmed that FaR consistently maintains distinct identities. While baseline models often merged two different subjects or lost specific markings, FaR kept identities sharp and separate, even when the subjects were of the same category.
Discussion and Limitations
Key Insights
- Compositional Training: Training on single-subject images is insufficient for multi-subject tasks. Synthetically fusing subjects during training is essential for teaching the model the concept of "co-existence."
- Attention as a Boundary: Attribute leakage is fundamentally a failure of spatial attention. Using segmentation masks to constrain attention maps effectively preserves identity integrity.
- Descriptive Class Anchoring: Using specific prompts (e.g.,
[S1] border collie doginstead of[S1] dog) allows the model to leverage pre-existing knowledge to better anchor new identities.
Limitations
Despite its success, FaR requires a pre-processing step involving segmentation and prior image generation, which adds to the initial training overhead. Additionally, while the framework excels with pairs of subjects, further research is needed to determine how the Localized Refinement loss scales as the number of unique concepts in a single frame increases. Future work may also explore more sophisticated blending techniques to improve the naturalness of lighting and shadows in the fused training images.
Summary updated