Introduction
Identity (ID) customization in text-to-image generation aims to create high-fidelity images of specific individuals based on a reference photo and a text prompt. Traditional methods often struggle with a trade-off between fidelity and efficiency. Existing solutions like IP-Adapter or PhotoMaker typically rely on external image encoders, such as CLIP, to extract features. However, these external features often suffer from a "modality gap"—they are not perfectly aligned with the diffusion model's internal latent space—leading to a loss of fine-grained facial details and structural integrity.
Bridging the Modality Gap with Inv-Adapter
Inv-Adapter introduces a paradigm shift by utilizing the diffusion model itself as the feature extractor. Instead of an external CLIP encoder, the framework employs DDIM image inversion to map the reference image into the model's own latent noise space.
By extracting intermediate "diffusion features" directly from the U-Net during a denoising pass, the model ensures that the injected information is perfectly aligned with its internal representations. This eliminates the modality gap and preserves subtle identity markers—like specific eye shapes or skin textures—that external encoders often miss or smooth over.
The Embedded Attention Adapter (EAA)
The core of the Inv-Adapter is the Embedded Attention Adapter (EAA), a lightweight module consisting of only 48 million parameters. Unlike previous adapters that primarily target cross-attention layers to handle semantic concepts, the EAA injects features into both self-attention and cross-attention layers.
As shown in the architecture above, the EAA uses a token aggregation strategy to balance the influence of image features and text prompts. Injecting features into self-attention layers is particularly crucial, as these layers govern the structural and geometric consistency of the face, ensuring the generated output maintains the unique identity of the reference person while adhering to the user's descriptive prompt.
Performance and Efficiency
Inv-Adapter achieves state-of-the-art results across multiple benchmarks, including CelebA-HQ and FFHQ.
- High Fidelity: It outperforms competitors like InstantID and PhotoMaker in facial similarity (FACE-SIM) and feature similarity (DINO) metrics.
- Superior Quality: Lower FID scores indicate more realistic and higher-quality image generation compared to existing methods.
- Efficiency: With a footprint of just 48M trainable parameters, it is significantly more lightweight than InstantID (>1024M) or PhotoMaker (460M), making it highly efficient for deployment.
Conclusion
Inv-Adapter demonstrates that a diffusion model's internal features are its own best guide for identity preservation. By replacing heavy external encoders with a lightweight inversion-based adapter, the framework achieves superior fidelity and efficiency. While the DDIM inversion process currently adds some computational overhead, future iterations utilizing acceleration techniques like Latent Consistency Models (LCM) promise to make this high-fidelity customization even more accessible.
Summary updated