DP-Adapter: Dual-Pathway Adapter for Boosting Fidelity and Text Consistency in Customizable Human Image Generation

Ye Wang, Xuping Xie, Lanjun Wang, Zili Yi, Rui Ma

TL;DR

DP-Adapter introduces a dual-pathway framework for personalized text-to-image generation that eliminates the typical tug-of-war between identity preservation and text prompt adherence. By decoupling the generation process into face-sensitive and text-sensitive regions, it achieves state-of-the-art facial fidelity while accurately rendering complex textual descriptions.

What Problem Does This Paper Address?

Customized human image generation—creating novel images of a specific person based on a reference photo and a text prompt—typically suffers from a frustrating trade-off. Existing tuning-free models either excel at keeping the person's face accurate but ignore text prompts (e.g., failing to render a requested "Superman outfit"), or they follow the text prompt well but lose the person's unique facial identity.

The authors identify the root cause as a lack of spatial constraints when combining visual and textual prompts. This leads to mutual interference: the visual identity "bleeds" into the background and clothing, or the textual prompt overwrites the facial features.

Figure 1: Compared to other methods, our approach excels in preserving human identities while ensuring textual se- mantic consistency.

Methods

The authors propose the Dual-Pathway Adapter (DP-Adapter), which separates image generation into two distinct streams based on spatial sensitivity:

  1. Identity-Enhancing Adapter (IEA): Dedicated to visually sensitive regions (the face). It uses a region-based denoising loss constrained by a face mask to ensure high-fidelity facial reconstruction.
  2. Textual-Consistency Adapter (TCA): Dedicated to text-sensitive regions (background, clothing, poses). To prevent the reference image from overriding the text prompt, it employs a "softened visual injection" by lowering the image prompt attention weight ($\alpha = 0.5$).
Figure 2: The DP-Adapter framework utilizes a region-separate processing approach, dividing the target image generation into visually sensitive and text-sensitive regions. For visually sensitive regions, an identity-enhancing adapter is employed to ensure high-fidelity face generation. For text-sensitive regions, a textual-consistency adapter is used to maintain semantic accuracy. To seamlessly blend these regions, we introduce a fine-grained feature-level blending module, which effectively integrates the different areas and minimizes artifacts. For simplicity, the textual injection branch is not displayed.

To merge these pathways seamlessly without creating harsh edges or visual artifacts, they introduce a Fine-Grained Feature-Level Blending (FFB) module. Instead of blending in the noisy latent space (which typically causes artifacts), FFB combines hierarchical semantic features directly within the U-Net blocks using a mask.

Automatic Mask Generation

Since masks aren't available during inference, DP-Adapter extracts cross-attention maps from the TCA, applies thresholding and largest-region filtering, and binarizes the result to automatically generate a precise face mask on the fly.

Figure 3: The process of generating face mask images during the inference stage.

Datasets

  • Training: Trained on the FFHQ (Flickr-Faces-HQ) dataset, preprocessed by FastComposer.
  • Evaluation: Evaluated on a curated test set of 30 individual image samples (balanced across gender and age) combined with 40 diverse text prompts (covering actions, clothing, accessories, and scenes), resulting in a total of 4.8K generated evaluation images.

Key Insights and Ideas

  • Modality Separation by Region: Visual features and text prompts shouldn't compete globally. Visual features belong in the face region, while text prompts should dominate the background and clothing.
  • Softening vs. Dropping: Completely removing the image prompt in text-sensitive regions hurts overall image coherence. However, "softening" it provides just enough context without overriding text semantics.
  • Semantic Blending Over Latent Blending: Traditional diffusion blending in the latent noise space lacks semantic awareness, causing blending artifacts. Blending hierarchically within the U-Net feature layers preserves structural and contextual harmony.
  • Cross-Attention Maps as Layout Priors: The model's internal text-to-image cross-attention maps inherently know where the face is, allowing for zero-shot automatic mask generation during inference without needing external face-detection models.

Key Results

  • Quantitative Superiority: DP-Adapter achieves a state-of-the-art Face Score of 81.06 (outperforming InstantID's 79.91 and IP-Adapter's 80.78) while maintaining a high CLIP-IT text similarity score of 25.07 (significantly beating InstantID's 22.64). It also scored highest on human preference metrics (PickScore: 21.97, HPS: 21.31).
  • Qualitative Adherence: It successfully handles complex multi-category prompts (e.g., "A woman wearing a Superman outfit and holding a bunch of roses in the autumn tree") where previous methods failed to render the outfit or background accurately while keeping the face intact.
  • Downstream Applications: The architecture naturally supports downstream tasks like controllable headshot-to-full-body generation, age editing, expression editing, and animating historical figures from old photos.

Limitations and Future Work

  • Artistic Styles: The model faces limitations when handling corner cases where the text prompt demands specific artistic or non-realistic styles (e.g., oil paintings or sketches), as the identity-enhancing pathway is heavily biased toward photographic realism.
  • Future Directions: The authors plan to extend this dual-pathway approach to general subject-driven synthesis (e.g., objects, pets) and improve robustness in highly challenging stylistic scenarios.

Summary updated

Abstract

With the growing popularity of personalized human content creation and sharing, there is a rising demand for advanced techniques in customized human image generation. However, current methods struggle to simultaneously maintain the fidelity of human identity and ensure the consistency of textual prompts, often resulting in suboptimal outcomes. This shortcoming is primarily due to the lack of effective constraints during the simultaneous integration of visual and textual prompts, leading to unhealthy mutual interference that compromises the full expression of both types of input. Building on prior research that suggests visual and textual conditions influence different regions of an image in distinct ways, we introduce a novel Dual-Pathway Adapter (DP-Adapter) to enhance both high-fidelity identity preservation and textual consistency in personalized human image generation. Our approach begins by decoupling the target human image into visually sensitive and text-sensitive regions. For visually sensitive regions, DP-Adapter employs an Identity-Enhancing Adapter (IEA) to preserve detailed identity features. For text-sensitive regions, we introduce a Textual-Consistency Adapter (TCA) to minimize visual interference and ensure the consistency of textual semantics. To seamlessly integrate these pathways, we develop a Fine-Grained Feature-Level Blending (FFB) module that efficiently combines hierarchical semantic features from both pathways, resulting in more natural and coherent synthesis outcomes. Additionally, DP-Adapter supports various innovative applications, including controllable headshot-to-full-body portrait generation, age editing, old-photo to reality, and expression editing.
BibTeX
@inproceedings{wang2025dp,
  author = {Ye Wang and Xuping Xie and Lanjun Wang and Zili Yi and Rui Ma},
  title = {{DP-Adapter: Dual-Pathway Adapter for Boosting Fidelity and Text Consistency in Customizable Human Image Generation}},
  booktitle = {Graph. Model},
  year = {2025},
  url = {https://openreview.net/forum?id=v22U3OC2m2},
}
Figure