Inv-Adapter: ID Customization Generation via Image Inversion and Lightweight Parameter Adapter

Peng Xing, Ning Wang, Jianbo Ouyang, Zechao LiIEEE TPAMI 2025 · DOI

Introduction

Identity (ID) customization in text-to-image generation aims to create high-fidelity images of specific individuals based on a reference photo and a text prompt. Traditional methods often struggle with a trade-off between fidelity and efficiency. Existing solutions like IP-Adapter or PhotoMaker typically rely on external image encoders, such as CLIP, to extract features. However, these external features often suffer from a "modality gap"—they are not perfectly aligned with the diffusion model's internal latent space—leading to a loss of fine-grained facial details and structural integrity.

Bridging the Modality Gap with Inv-Adapter

Inv-Adapter introduces a paradigm shift by utilizing the diffusion model itself as the feature extractor. Instead of an external CLIP encoder, the framework employs DDIM image inversion to map the reference image into the model's own latent noise space.

Comparison between the traditional CLIP-based scheme and the proposed Inv-Adapter approach.

By extracting intermediate "diffusion features" directly from the U-Net during a denoising pass, the model ensures that the injected information is perfectly aligned with its internal representations. This eliminates the modality gap and preserves subtle identity markers—like specific eye shapes or skin textures—that external encoders often miss or smooth over.

The Embedded Attention Adapter (EAA)

The core of the Inv-Adapter is the Embedded Attention Adapter (EAA), a lightweight module consisting of only 48 million parameters. Unlike previous adapters that primarily target cross-attention layers to handle semantic concepts, the EAA injects features into both self-attention and cross-attention layers.

Detailed architecture of the Inv-Adapter, highlighting the Feature Extraction, Embedded Attention Adapter, and Customized Generation stages.

As shown in the architecture above, the EAA uses a token aggregation strategy to balance the influence of image features and text prompts. Injecting features into self-attention layers is particularly crucial, as these layers govern the structural and geometric consistency of the face, ensuring the generated output maintains the unique identity of the reference person while adhering to the user's descriptive prompt.

Performance and Efficiency

Inv-Adapter achieves state-of-the-art results across multiple benchmarks, including CelebA-HQ and FFHQ.

  • High Fidelity: It outperforms competitors like InstantID and PhotoMaker in facial similarity (FACE-SIM) and feature similarity (DINO) metrics.
  • Superior Quality: Lower FID scores indicate more realistic and higher-quality image generation compared to existing methods.
  • Efficiency: With a footprint of just 48M trainable parameters, it is significantly more lightweight than InstantID (>1024M) or PhotoMaker (460M), making it highly efficient for deployment.

Conclusion

Inv-Adapter demonstrates that a diffusion model's internal features are its own best guide for identity preservation. By replacing heavy external encoders with a lightweight inversion-based adapter, the framework achieves superior fidelity and efficiency. While the DDIM inversion process currently adds some computational overhead, future iterations utilizing acceleration techniques like Latent Consistency Models (LCM) promise to make this high-fidelity customization even more accessible.

Summary updated

BibTeX
@article{xing2025inv,
  author = {Peng Xing and Ning Wang and Jianbo Ouyang and Zechao Li},
  title = {{Inv-Adapter: ID Customization Generation via Image Inversion and Lightweight Parameter Adapter}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3590321},
}

Loading related content...

Figure