Abstract
Personalizing text-to-image models using adapters often suffers from "shortcut learning," where the model captures extraneous features like pose, lighting, and background instead of the target attribute (e.g., identity). We present Shortcut-Rerouted (SR) Adapter Training, a method that explicitly provides these confounding factors through auxiliary modules during training. By "explaining away" these shortcuts, the primary adapter is forced to learn a disentangled representation of the target attribute. At inference, the auxiliary modules are removed, enabling high-fidelity identity preservation with superior prompt adherence.
1. Introduction
The rise of adapters like IP-Adapter and LoRA has enabled efficient personalization of large-scale text-to-image models. However, these methods typically rely on a single-image reconstruction objective. When a model is tasked with reconstructing a specific image of a subject, it often takes the path of least resistance, learning to replicate the entire image—including specific poses and environmental conditions—rather than isolating the subject's identity. This leads to entanglement, where the adapter overrides text prompts during inference, failing to generate the subject in new contexts or poses.
2. The Shortcut Problem in Adapter Training
Neural networks are inherently biased toward learning the simplest features that satisfy the loss function. In the context of identity adapters, a single reference image contains both "identity-relevant" features (facial structure, head shape) and "identity-irrelevant" features (lighting, expression, pose). Because the reconstruction loss treats all pixels equally, the adapter frequently learns to encode the entire scene. This "shortcut" prevents the model from understanding the abstract concept of the person, resulting in poor prior preservation and limited controllability.
3. Shortcut-Rerouted (SR) Training
To address this, we introduce Shortcut-Rerouted (SR) Adapter Training. The core principle is to provide a dedicated channel for the features we do not want the adapter to learn.
3.1 The "Explain It Away" Principle
By introducing an auxiliary "Shortcut Module" ($\mathcal{S}_C$) that handles confounding factors, we satisfy the reconstruction loss for those specific attributes. This forces the main identity adapter to focus exclusively on the remaining information—the target identity. This is a classic concept in computer vision applied to generative adapters: if one part of the architecture handles the "noise," the other part is forced to find the "signal."
3.2 Implementation Strategies
We explore two primary implementations of this rerouting:
- SR-CN (ControlNet): A pre-trained ControlNet is used during training to provide pose and expression maps directly to the generator. Since the generator can rely on the ControlNet for structural guidance, the identity adapter has no incentive to encode pose information.
- SR-LoRA: To handle domain gaps between the foundation model and specific training datasets, a LoRA is trained to "absorb" the dataset's specific stylistic or distributional shortcuts.
4. Experimental Setup
The method was trained on a curated dataset of several million high-quality human images, captioned using advanced multimodal models (Qwen2.5-14B and InternViT-300M). Evaluation focused on facial and full-body personalization, comparing SR-Adapter against strong baselines such as InfU, PuLID, and standard IP-Adapter.
5. Results and Discussion
SR-Rerouted adapters demonstrate significant improvements in both quantitative metrics and qualitative flexibility.
5.1 Quantitative Performance
In facial adapter benchmarks, the SR-CN method achieved superior Head Pose control (12.67 compared to 17.71 for InfU) and improved Prior Preservation (LPIPS of 0.39 vs. 0.48 for standard IP-Adapter). These metrics suggest that the model successfully decoupled identity from the reference image's structure.
5.2 Qualitative Flexibility
Qualitatively, SR-trained models exhibit remarkable prompt adherence. While baseline models often struggle to change the expression of a reference subject (e.g., failing to generate a "snarling" face if the reference is smiling), the SR-Adapter allows for full expressive range while maintaining identity. This extends to full-body control, where the subject's stance can be modified via text without interference from the original training pose.
5.3 LLM-as-a-Judge
Traditional metrics like FaceNet often fail to capture holistic identity features like hair and head shape. The authors advocate for the use of multimodal LLMs (e.g., GPT-4o) to provide a more human-centric evaluation of identity fidelity and expression accuracy, which aligns more closely with user expectations.
6. Conclusion
Shortcut Rerouting offers a principled approach to training disentangled adapters. By explicitly providing the shortcuts we wish to avoid, we force the model to learn the underlying signal rather than the noise. While the current study focuses on encoder-based adapters, future work will explore applying this framework to LoRA-based identity training and more complex architectural baselines. As with all personalization research, the authors acknowledge the ethical risks regarding deepfakes and the need for responsible deployment.
Summary updated