X- Adapter: Universal Compatibility of Plugins for Upgraded Diffusion Model

Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, Mike Zheng Shou

TL;DR

X-Adapter is a universal "upgrader" that allows pretrained plugins (like ControlNet and LoRA) designed for older diffusion models (e.g., Stable Diffusion 1.5) to work seamlessly with newer, more powerful models (e.g., SDXL). By training a small adapter network instead of retraining every individual plugin, it preserves the massive ecosystem of community-built tools while leveraging the superior image quality of upgraded foundational models.

What Problem Does This Paper Address?

The rapid evolution of text-to-image diffusion models creates a significant "ecosystem lag." While foundational models like SDXL offer much higher resolution and better prompt adherence than their predecessors (like SD 1.5), the community has spent thousands of GPU hours developing specialized "plugins"—ControlNets for structural guidance, LoRAs for specific styles, and adapters for image editing.

When a new base model is released, these plugins typically become obsolete because of:

  1. Dimension Mismatches: The internal feature sizes of the new model don't match the old plugins.
  2. Latent Space Gaps: Different models operate in different mathematical "latent" spaces.
  3. Architectural Changes: The "connectors" where plugins attach to the network often disappear or move in upgraded versions.

Retraining every plugin for every new base model is prohibitively expensive. X-Adapter aims to solve this by creating a single, universal bridge between the old plugin ecosystem and new foundational models.

Methods

The authors propose X-Adapter, which acts as a controller for the upgraded model. Its architecture and training involve three clever components:

  • The Frozen Anchor: X-Adapter keeps a completely frozen copy of the original base model (SD 1.5). This ensures that all existing plugins can still "plug in" to the exact architecture they were designed for.
  • Trainable Mapping Layers: To bridge the gap, the authors add small, trainable mapping layers (stacks of ResNet blocks) that take features from the old model’s decoder and remap them to fit the dimensions and feature space of the new model’s decoder.
  • Null-Text Training Strategy: During training, the upgraded model (SDXL) is given an empty text prompt (""), while the X-Adapter is given the actual descriptive prompt. This forces the upgraded model to rely entirely on the X-Adapter’s guidance to generate the image, effectively teaching it how to "listen" to the adapter.
  • Two-Stage Denoising: At inference time, the latent spaces of the two models are often unaligned. The authors use a two-stage approach: they start the generation process using the old model for a "warmup" (about 20% of the steps) to establish the basic structure, then switch to the upgraded model via X-Adapter to refine the details and quality.

Datasets

  • Training: The authors used a subset of LAION-high-resolution containing 300,000 image-text pairs.
  • Evaluation:
    • COCO Validation Set: 5,000 images were used to test ControlNet-based guidance (Canny and Depth).
  • Civitai Prompts: 20 prompts were selected for various LoRAs to test style retention.
  • User Study: 5 users ranked generations based on image quality and fidelity to the original plugin's intent.

Key Insights and Ideas

  • Preservation over Adaptation: Instead of trying to force a plugin to fit a new model, X-Adapter preserves the plugin's native environment (the old model) and focuses on translating the output of that environment into a language the new model understands.
  • The Power of "Remixing": Because X-Adapter maintains the connectors of both models, it enables "Plugin Remixing." For example, you can use a ControlNet designed for SD 1.5 alongside a LoRA designed for SDXL, bridging two different generations of AI tools in a single workflow.
  • Guidance via Addition: Through ablation studies, the authors found that simply adding the remapped features to the new model's decoder was more effective than more complex fusion methods like SPADE. This suggests that the mapping layers are successfully learning the "offset" between the two models' internal representations.

Key Results

  • Quantitative Superiority: In tests with SDXL, X-Adapter achieved a CLIP-score of 0.2632 (measuring prompt alignment), outperforming both the original SD 1.5 (0.2426) and the SDEdit baseline (0.2594).
  • Style and Control: For LoRA plugins, X-Adapter achieved a Style Similarity of 0.83, significantly higher than the SDEdit baseline (0.72), proving it better preserves the specific aesthetic the plugin was meant to provide.
  • Visual Quality: Qualitative results show that X-Adapter produces images with the high-frequency details and resolution characteristic of SDXL, but with the precise structural control (e.g., specific poses or edge maps) provided by SD 1.5-era ControlNets.

Limitations and Future Work

The authors acknowledge a primary limitation regarding Identity Preservation. Plugins that rely on modifying the text-encoder (like IP-Adapter for face/identity consistency) don't always translate perfectly through X-Adapter. This is because X-Adapter primarily focuses on remapping the feature space of the UNet decoders rather than the text-embedding space.

Future work may involve expanding the adapter's reach to include these text-encoder-based concepts, potentially creating an even more "universal" compatibility layer.

Summary updated

Abstract

We introduce X-Adapter, a universal upgrader to enable the pretrained plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the upgraded text-to-image diffusion model (e.g., SDXL) without further retraining. We achieve this goal by training an additional network to control the frozen upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a frozen copy of the old model to preserve the connectors of different plugins. Additionally, X-Adapter adds trainable mapping layers that bridge the decoders from models of different versions for feature remapping. The remapped features will be used as guidance for the upgraded model. To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model. After training, we also introduce a two-stage denoising strategy to align the initial latents of X Adapter and the upgraded model. Thanks to our strategies, X-Adapter demonstrates universal compatibility with various plugins and also enables plugins of different versions to work together, thereby expanding the functionalities of diffusion community. To verify the effectiveness of the proposed method, we conduct extensive experiments and the results show that X-Adapter may facilitate wider application in the upgraded foundational diffusion model. Project page at: https://showlab.github.io/X-Adapter/.
BibTeX
@inproceedings{ran2024x,
  author = {Lingmin Ran and Xiaodong Cun and Jia-Wei Liu and Rui Zhao and Song Zijie and Xintao Wang and Jussi Keppo and Mike Zheng Shou},
  title = {{X- Adapter: Universal Compatibility of Plugins for Upgraded Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=jAb1FCES9r},
}

Loading related content...

Figure