Preventing Shortcuts in Adapter Training via Providing the Shortcuts

Anujraaj Goyal, Guocheng Qian, Huseyin Coskun, Aarush Gupta, Himmy Tam, Daniil Ostashev, Ju Hu, Dhritiman Sagar, Sergey Tulyakov, Kfir Aberman, Kuan-Chieh Wang

Abstract

Personalizing text-to-image models using adapters often suffers from "shortcut learning," where the model captures extraneous features like pose, lighting, and background instead of the target attribute (e.g., identity). We present Shortcut-Rerouted (SR) Adapter Training, a method that explicitly provides these confounding factors through auxiliary modules during training. By "explaining away" these shortcuts, the primary adapter is forced to learn a disentangled representation of the target attribute. At inference, the auxiliary modules are removed, enabling high-fidelity identity preservation with superior prompt adherence.

1. Introduction

The rise of adapters like IP-Adapter and LoRA has enabled efficient personalization of large-scale text-to-image models. However, these methods typically rely on a single-image reconstruction objective. When a model is tasked with reconstructing a specific image of a subject, it often takes the path of least resistance, learning to replicate the entire image—including specific poses and environmental conditions—rather than isolating the subject's identity. This leads to entanglement, where the adapter overrides text prompts during inference, failing to generate the subject in new contexts or poses.

2. The Shortcut Problem in Adapter Training

Neural networks are inherently biased toward learning the simplest features that satisfy the loss function. In the context of identity adapters, a single reference image contains both "identity-relevant" features (facial structure, head shape) and "identity-irrelevant" features (lighting, expression, pose). Because the reconstruction loss treats all pixels equally, the adapter frequently learns to encode the entire scene. This "shortcut" prevents the model from understanding the abstract concept of the person, resulting in poor prior preservation and limited controllability.

3. Shortcut-Rerouted (SR) Training

To address this, we introduce Shortcut-Rerouted (SR) Adapter Training. The core principle is to provide a dedicated channel for the features we do not want the adapter to learn.

3.1 The "Explain It Away" Principle

By introducing an auxiliary "Shortcut Module" ($\mathcal{S}_C$) that handles confounding factors, we satisfy the reconstruction loss for those specific attributes. This forces the main identity adapter to focus exclusively on the remaining information—the target identity. This is a classic concept in computer vision applied to generative adapters: if one part of the architecture handles the "noise," the other part is forced to find the "signal."

3.2 Implementation Strategies

We explore two primary implementations of this rerouting:

  • SR-CN (ControlNet): A pre-trained ControlNet is used during training to provide pose and expression maps directly to the generator. Since the generator can rely on the ControlNet for structural guidance, the identity adapter has no incentive to encode pose information.
  • SR-LoRA: To handle domain gaps between the foundation model and specific training datasets, a LoRA is trained to "absorb" the dataset's specific stylistic or distributional shortcuts.

4. Experimental Setup

The method was trained on a curated dataset of several million high-quality human images, captioned using advanced multimodal models (Qwen2.5-14B and InternViT-300M). Evaluation focused on facial and full-body personalization, comparing SR-Adapter against strong baselines such as InfU, PuLID, and standard IP-Adapter.

5. Results and Discussion

SR-Rerouted adapters demonstrate significant improvements in both quantitative metrics and qualitative flexibility.

5.1 Quantitative Performance

In facial adapter benchmarks, the SR-CN method achieved superior Head Pose control (12.67 compared to 17.71 for InfU) and improved Prior Preservation (LPIPS of 0.39 vs. 0.48 for standard IP-Adapter). These metrics suggest that the model successfully decoupled identity from the reference image's structure.

5.2 Qualitative Flexibility

Qualitatively, SR-trained models exhibit remarkable prompt adherence. While baseline models often struggle to change the expression of a reference subject (e.g., failing to generate a "snarling" face if the reference is smiling), the SR-Adapter allows for full expressive range while maintaining identity. This extends to full-body control, where the subject's stance can be modified via text without interference from the original training pose.

5.3 LLM-as-a-Judge

Traditional metrics like FaceNet often fail to capture holistic identity features like hair and head shape. The authors advocate for the use of multimodal LLMs (e.g., GPT-4o) to provide a more human-centric evaluation of identity fidelity and expression accuracy, which aligns more closely with user expectations.

6. Conclusion

Shortcut Rerouting offers a principled approach to training disentangled adapters. By explicitly providing the shortcuts we wish to avoid, we force the model to learn the underlying signal rather than the noise. While the current study focuses on encoder-based adapters, future work will explore applying this framework to LoRA-based identity training and more complex architectural baselines. As with all personalization research, the authors acknowledge the ethical risks regarding deepfakes and the need for responsible deployment.

Summary updated

Abstract

Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt. In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.
BibTeX
@inproceedings{goyal2025preventing,
  author = {Anujraaj Goyal and Guocheng Qian and Huseyin Coskun and Aarush Gupta and Himmy Tam and Daniil Ostashev and Ju Hu and Dhritiman Sagar and Sergey Tulyakov and Kfir Aberman and Kuan-Chieh Wang},
  title = {{Preventing Shortcuts in Adapter Training via Providing the Shortcuts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=cZMno8E3yp},
}
Figure