Nested Attention: Semantic-aware Attention Values for Concept Personalization

Or Patashnik, Rinon Gal, Daniil Ostashev, Sergey Tulyakov, Kfir Aberman, Daniel Cohen-Or

Abstract

Concept personalization in text-to-image models often faces a fundamental trade-off between identity fidelity and prompt adherence. This paper introduces Nested Attention, a novel mechanism that dynamically generates query-dependent attention values for a single text token. By injecting high-fidelity visual details through a specific token anchor, the method preserves the model's learned prior while capturing intricate subject details. Our approach enables high-fidelity personalization across diverse styles and supports multi-subject generation without architectural modifications.

Introduction

Personalizing text-to-image models—allowing users to generate images of specific subjects in new contexts—is a significant research challenge. Current methods typically fall into two camps: single-token optimization, which often lacks the bandwidth for intricate details, and multi-token encoder-based injection, which can overwhelm the model's prior and lead to poor prompt alignment. Nested Attention addresses this "tug-of-war" by drastically increasing a single token's expressiveness through a nested attention layer.

Methodology

The Nested Attention Mechanism

In standard cross-attention, a text token (e.g., "person") is mapped to a static Value vector. Every image patch attending to that token receives the same representation. Nested Attention replaces this static vector with a dynamic, query-dependent output:

  1. Rich Encoding: An encoder (CLIP combined with a trained Q-Former) extracts a multi-token representation from the reference image.
  2. Query-Dependent Values: When the diffusion model generates a specific spatial patch (e.g., an eye or hair), that spatial query is passed into an internal "nested" attention layer. This layer attends to the rich visual tokens to produce a custom Value vector tailored to that specific patch.
  3. Semantic Routing: This process naturally learns to route features; for instance, a "nose" query in the diffusion model weights the corresponding nose features from the reference tokens.

Training and Regularization

During training, the base Stable Diffusion XL (SDXL) model remains frozen. Only the image encoder and nested attention layers are optimized. To prevent high-fidelity values from dominating the generation, a regularization technique bounds the mathematical norm of the new values to match original word embeddings.

Multi-Subject and Stylized Generation

Because the mechanism anchors visual features to specific words, the model's original layout and stylistic capabilities remain intact. This allows for seamless multi-subject generation by tying different nested attention layers to different words in a prompt (e.g., "a [person] and their [dog]").

Multi-subject generation on a beach

Furthermore, the "Single-Token Anchoring" ensures that stylistic modifiers (e.g., "watercolor," "cubism") function as intended. Unlike decoupled-attention methods that often revert to photorealism, Nested Attention maintains identity even in highly abstract or stylized contexts.

Stylized 3D figurine generation

Results and Evaluation

Quantitative Performance

Nested Attention demonstrates superior performance on Pareto curves mapping Identity Similarity (via face recognition) against Text Similarity (via CLIP). It strictly dominates established baselines including IP-Adapter, InstantID, and PhotoMaker.

User Preference

In blind A/B testing, users preferred Nested Attention's outputs significantly over competitors: 71% over IP-Adapter (Face), 66% over InstantID, and 56% over LCM-Lookahead.

Inference-Time Control

Users can modulate the trade-off between identity and editability at inference time by scaling the attention score of the personalized token, providing a flexible control mechanism similar to adapter scales.

Conclusion and Future Work

Nested Attention provides a robust framework for high-fidelity concept personalization. While highly effective, the method currently faces challenges with "attention leakage" when generating multiple subjects of the same domain (e.g., two different people). Future work will explore domain-agnostic encoders and applications in subject-conditioned inpainting and style transfer.

Summary updated

BibTeX
@inproceedings{patashnik2025nested,
  author = {Or Patashnik and Rinon Gal and Daniil Ostashev and Sergey Tulyakov and Kfir Aberman and Daniel Cohen-Or},
  title = {{Nested Attention: Semantic-aware Attention Values for Concept Personalization}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730634},
}
Figure