Preventing Shortcuts in Adapter Training via Providing the Shortcuts

Anujraaj Goyal, Guocheng Gordon Qian, Huseyin Coskun, Aarush Gupta, Himmy Tam, Daniil Ostashev, Ju Hu, Dhritiman Sagar, Sergey Tulyakov, Kfir Aberman, Kuan-Chieh Jackson Wang

Abstract

Personalizing text-to-image models using adapters often suffers from "shortcut learning," where the model captures extraneous features like pose, lighting, and background instead of the target attribute (e.g., identity). We present Shortcut-Rerouted (SR) Adapter Training, a method that explicitly provides these confounding factors through auxiliary modules during training. By "explaining away" these shortcuts, the primary adapter is forced to learn a disentangled representation of the target attribute. At inference, the auxiliary modules are removed, enabling high-fidelity identity preservation with superior prompt adherence.

1. Introduction

The rise of adapters like IP-Adapter and LoRA has enabled efficient personalization of large-scale text-to-image models. However, these methods typically rely on a single-image reconstruction objective. When a model is tasked with reconstructing a specific image of a subject, it often takes the path of least resistance, learning to replicate the entire image—including specific poses and environmental conditions—rather than isolating the subject's identity. This leads to entanglement, where the adapter overrides text prompts during inference, failing to generate the subject in new contexts or poses.

2. The Shortcut Problem in Adapter Training

Neural networks are inherently biased toward learning the simplest features that satisfy the loss function. In the context of identity adapters, a single reference image contains both "identity-relevant" features (facial structure, head shape) and "identity-irrelevant" features (lighting, expression, pose). Because the reconstruction loss treats all pixels equally, the adapter frequently learns to encode the entire scene. This "shortcut" prevents the model from understanding the abstract concept of the person, resulting in poor prior preservation and limited controllability.

3. Shortcut-Rerouted (SR) Training

To address this, we introduce Shortcut-Rerouted (SR) Adapter Training. The core principle is to provide a dedicated channel for the features we do not want the adapter to learn.

3.1 The "Explain It Away" Principle

By introducing an auxiliary "Shortcut Module" ($\mathcal{S}_C$) that handles confounding factors, we satisfy the reconstruction loss for those specific attributes. This forces the main identity adapter to focus exclusively on the remaining information—the target identity. This is a classic concept in computer vision applied to generative adapters: if one part of the architecture handles the "noise," the other part is forced to find the "signal."

3.2 Implementation Strategies

We explore two primary implementations of this rerouting:

  • SR-CN (ControlNet): A pre-trained ControlNet is used during training to provide pose and expression maps directly to the generator. Since the generator can rely on the ControlNet for structural guidance, the identity adapter has no incentive to encode pose information.
  • SR-LoRA: To handle domain gaps between the foundation model and specific training datasets, a LoRA is trained to "absorb" the dataset's specific stylistic or distributional shortcuts.

4. Experimental Setup

The method was trained on a curated dataset of several million high-quality human images, captioned using advanced multimodal models (Qwen2.5-14B and InternViT-300M). Evaluation focused on facial and full-body personalization, comparing SR-Adapter against strong baselines such as InfU, PuLID, and standard IP-Adapter.

5. Results and Discussion

SR-Rerouted adapters demonstrate significant improvements in both quantitative metrics and qualitative flexibility.

5.1 Quantitative Performance

In facial adapter benchmarks, the SR-CN method achieved superior Head Pose control (12.67 compared to 17.71 for InfU) and improved Prior Preservation (LPIPS of 0.39 vs. 0.48 for standard IP-Adapter). These metrics suggest that the model successfully decoupled identity from the reference image's structure.

5.2 Qualitative Flexibility

Qualitatively, SR-trained models exhibit remarkable prompt adherence. While baseline models often struggle to change the expression of a reference subject (e.g., failing to generate a "snarling" face if the reference is smiling), the SR-Adapter allows for full expressive range while maintaining identity. This extends to full-body control, where the subject's stance can be modified via text without interference from the original training pose.

5.3 LLM-as-a-Judge

Traditional metrics like FaceNet often fail to capture holistic identity features like hair and head shape. The authors advocate for the use of multimodal LLMs (e.g., GPT-4o) to provide a more human-centric evaluation of identity fidelity and expression accuracy, which aligns more closely with user expectations.

6. Conclusion

Shortcut Rerouting offers a principled approach to training disentangled adapters. By explicitly providing the shortcuts we wish to avoid, we force the model to learn the underlying signal rather than the noise. While the current study focuses on encoder-based adapters, future work will explore applying this framework to LoRA-based identity training and more complex architectural baselines. As with all personalization research, the authors acknowledge the ethical risks regarding deepfakes and the need for responsible deployment.

Summary updated

BibTeX
@inproceedings{goyal2025preventing,
  author = {Anujraaj Goyal and Guocheng Gordon Qian and Huseyin Coskun and Aarush Gupta and Himmy Tam and Daniil Ostashev and Ju Hu and Dhritiman Sagar and Sergey Tulyakov and Kfir Aberman and Kuan-Chieh Jackson Wang},
  title = {{Preventing Shortcuts in Adapter Training via Providing the Shortcuts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=cZMno8E3yp},
}
Figure