The Core Dilemma of Personalization
When we personalize a text-to-image diffusion model—teaching it a specific subject like a unique pet or a custom character—we expect it to seamlessly blend this new concept into any imaginative scenario. However, as synthesized in Personalized Image Generation with Deep Generative Models: A Decade Survey, text-to-image personalization introduces a fundamental tension: the model either captures the subject's fine details perfectly but ignores the prompt (overfitting), or it follows the prompt beautifully but loses the subject's unique identity (underfitting).
This phenomenon is known as the fidelity-editability trade-off. To build a truly controllable personalized model, researchers must find ways to inject specific visual identities without destroying the model's underlying creative flexibility.
Why the Attention Hijack Happens
To understand why this trade-off exists, we must look at how diffusion models process text instructions. In a standard model, text tokens guide image generation via cross-attention mechanisms, mapping words to specific spatial regions during the denoising process. When we introduce a custom concept via optimization (such as Textual Inversion or DreamBooth fine-tuning), we force a new token embedding or weight set into the model.
In Mitigating Semantic Collapse in Generative Personalization with Test-Time Embedding Adjustment, Anh Tuan Bui et al. identify this as the Semantic Collapsing Problem. During optimization, the learned token embedding often drifts geometrically, growing disproportionately large in magnitude. Because its vector norm becomes massive compared to standard text tokens, it dominates the cross-attention maps during inference.
The model effectively suffers an "attention hijack," focusing entirely on recreating the exact pixels of the reference image and drowning out the surrounding prompt instructions. As a result, as observed by Xulu Zhang et al. in Compositional Inversion for Stable Diffusion Models, the custom concept fails to compose with other elements, rigidly copying the original pose or background instead of adapting to the new context.
Restricting the Search Space
One way to prevent this attention hijack is to mathematically constrain the token embeddings so they cannot overpower the rest of the prompt. In Directional Textual Inversion for Personalized Text-to-Image Generation, Kunhee Kim et al. propose fixing the embedding magnitude to a natural scale on a hypersphere, optimizing only its direction. By preventing the vector norm from growing out-of-distribution, the token is forced to cooperate, rather than compete, with the rest of the prompt.
A parallel strategy involves limiting the model's internal capacity for change during fine-tuning. Instead of expanding the generation space, Shangyu Chen et al. introduce PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction. By restricting the learnable parameter space to half the size of standard low-rank adaptations (LoRA), the authors prevent the model from overfitting to the reference images, preserving its original creative flexibility and prompt obedience.
Disentangling Identity from Context
Another core issue is that standard personalization algorithms treat the reference image as a single monolithic concept, inadvertently binding the subject's identity to its original background, lighting, or pose. To break this entanglement, recent methods focus on explicit feature decoupling.
In CustomContrast: A Multilevel Contrastive Perspective for Subject-Driven Text-to-Image Customization, Nan Chen et al. shift from a self-reconstructive approach to a multilevel contrastive learning framework. This allows the model to isolate the subject's intrinsic identity from irrelevant contextual details.
Similarly, for tuning-free architectures that use image encoders, Siying Cui et al. develop IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models, which explicitly decouples identity-specific features from transient attributes like pose and expression during training. This ensures that the injected identity doesn't suppress environmental descriptions.
Exploiting the Time-Space Dimension
Diffusion models do not generate an image all at once; they build it iteratively over multiple denoising steps. Early steps establish global layout and geometry, while later steps fill in fine-grained textures and details. This temporal progression offers a natural lever for balancing fidelity and editability.
In A Neural Space-Time Representation for Text-to-Image Personalization, Yuval Alaluf et al. introduce Neural Space-Time Inversion (NeTI). Instead of learning a single static token, NeTI maps personalized concepts as a function of both denoising time and U-Net space. This provides a precise control knob: early, layout-heavy steps can prioritize prompt editability, while later, texture-heavy steps can prioritize visual fidelity.
Taking a reinforcement learning approach to this temporal division, Qiaoqiao Jin et al. propose Consis-GCPO: Consistency-Preserving Group Causal Preference Optimization for Vision Customization. By recognizing that text guides global structure early in the denoising process while visual references anchor fine details later, the authors apply time-sensitive optimizations that prevent the two signals from entering a destructive tug-of-war.
Directing Optimization via Preferences
Rather than relying purely on reconstruction losses, which inherently push the model toward overfitting to the reference pixels, researchers have turned to preference alignment.
In Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning, Yanting Miao et al. utilize Reward Preference Optimization (RPO). By employing a harmonic reward function that scores both subject fidelity and prompt alignment simultaneously, RPO guides the model toward a balanced Pareto frontier. This ensures the model satisfies both constraints without sacrificing one for the other.
Similarly, Daewon Chae et al. demonstrate in InstructBooth: Instruction-following Personalized Text-to-Image Generation that an explicit reinforcement learning fine-tuning stage can successfully prevent the model from forgetting how to execute complex actions and instructions when rendering a personalized subject.