DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis

Yongjin Choi, Chanhun Park, Seung Jun BaekAAAI 2025 · DOI

TL;DR

This paper introduces DynASyn, a novel method for personalizing multiple subjects within a single image for text-to-image generation. It tackles the challenge of maintaining subject identity while enabling diverse actions and interactions, outperforming existing methods in realism and flexibility.

What Problem Does This Paper Address?

Recent advancements in text-to-image (T2I) diffusion models have revolutionized image generation. However, personalizing these models to create images of specific subjects in novel contexts remains a challenge. While existing methods can personalize single subjects or even multiple subjects, they often struggle with:

  • Overfitting: Models can become too tied to the reference images, limiting their ability to generate diverse actions or poses. This is particularly problematic when only a single reference image is available.
  • Limited Action Diversity: Existing methods often fail to generate novel actions or dynamic interactions between subjects.
  • Multi-Subject Complexity: Personalizing multiple subjects simultaneously from a single image introduces further challenges in disentangling identities and controlling their actions independently.

DynASyn aims to address these limitations by enabling robust multi-subject personalization from a single image, focusing on preserving identity while allowing for dynamic action synthesis.

Methods

DynASyn introduces several key innovations to achieve its goals:

  • Concept-Based Attention Regularization: This technique leverages the rich prior knowledge embedded in large T2I models. It aligns subject appearances with class-specific priors by regularizing attention maps. Specifically, it uses a loss function (ICA loss) that compares attention maps generated from prompts with placeholders (e.g., <asset0>) to those generated from prompts with class names (e.g., "panda"). This helps prevent overfitting to the specific details of the reference image while retaining the subject's identity.
  • Concept-Based Prompt-and-Image Augmentation: To further enhance action diversity and overcome the limitations of single reference images, DynASyn employs a novel augmentation strategy. This involves:
  • Concise Description Generation: Using a vision-language model (VLM), it generates short, descriptive noun phrases for each subject in the input image.
  • Augmented Prompt Generation: A language model (LM) then creates diverse prompts describing new actions and interactions for these subjects.
  • Draft Image Generation: These augmented prompts are used with a T2I model to generate draft images.
  • Compose Augmented Images: The subjects from the draft images are then replaced with the original subjects, and any gaps are filled using inpainting. This creates augmented image-text pairs that capture novel scenarios.
  • Guided SDE Augmentation (GSA): Inspired by SDEdit, GSA uses a forward-backward diffusion process guided by the augmented prompts. This process helps align the augmented subject with the original identity while allowing for the depiction of new actions and poses.
  • Optimization with Augmented Data: The augmented prompts and images are used to further train the model. This optimization uses a loss function (LAUG) that focuses on inter-cross-attention without a reconstruction loss, preventing identity blending and promoting action learning.

The overall training objective is a weighted sum of masked diffusion loss (LMD), masked attention loss (LM2A), ICA loss, and the augmented data loss (LAUG), allowing for a balance between identity preservation and action diversity.

Datasets

The paper utilizes a combination of datasets for training and evaluation:

  • Training Data: A total of 15 image-text pairing datasets are used, including:
  • 7 individual datasets from Break-a-Scene.
  • 2 datasets from the COCO benchmark.
  • 2 datasets from Zhang et al.
  • 4 additionally collected proprietary datasets.
  • Augmentation Data: For the prompt-and-image augmentation, 30 text prompts per image are generated using GPT-4.

The paper does not introduce a new benchmark dataset but rather leverages existing ones and a novel augmentation strategy to create diverse training data.

Key Insights and Ideas

  • Leveraging T2I Priors for Personalization: The core idea is that large T2I models already contain vast knowledge about object concepts and typical behaviors. DynASyn effectively taps into this knowledge through attention map regularization to guide personalization, rather than solely relying on the limited information in reference images.
  • The Tension Between Identity and Diversity: The paper highlights the inherent trade-off between preserving subject identity and generating diverse actions. DynASyn's multi-faceted approach, combining attention regularization with prompt-image augmentation, is designed to navigate this tension effectively.
  • Augmentation as a Solution to Single-Image Limitations: By synthetically generating diverse prompt-image pairs from a single input, DynASyn overcomes the limitations of having only a few reference images, enabling more robust and flexible personalization.
  • Attention Maps as a Rich Source of Information: The method's reliance on attention maps for both regularization and optimization underscores their importance in understanding and controlling the diffusion process for personalized generation.

Key Results

  • Quantitative Performance: DynASyn achieves the highest text alignment (CLIP-T and Image Reward scores) across plain, action, and interaction sentence types compared to baseline methods like Textual Inversion (TI), DreamBooth (DB), and Break-a-Scene (B-a-S). While DB achieves a high identity score due to overfitting, DynASyn provides reasonable identity scores while excelling in text alignment and overall quality.
  • Qualitative Performance: DynASyn generates highly realistic images of subjects in novel contexts with dynamic interactions. Figure 4 and Figure 5 showcase its ability to depict subjects performing diverse actions and interacting with each other, often failing for baseline methods.
  • User Study Results: User interviews confirm DynASyn's superior performance, with participants rating its overall quality, text alignment, and identity preservation highly.

Limitations and Future Work

  • Dependence on T2I Backbone: DynASyn's performance is inherently tied to the quality and capabilities of the underlying T2I model (e.g., Stable Diffusion).
  • Challenging Prompts: The paper notes that models like Stable Diffusion can struggle with very challenging prompts, which can bottleneck the accurate alignment of subject actions with concept priors.
  • Future Directions:
    • Explore additional fine-tuning strategies to further refine the model's adaptability.
  • Investigate fine-tuning T2I models with high-quality, expansive datasets to overcome current limitations and facilitate more reliable and versatile image generation.

Summary updated

BibTeX
@inproceedings{choi2025dynasyn,
  author = {Yongjin Choi and Chanhun Park and Seung Jun Baek},
  title = {{DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32259},
}

Loading related content...

Figure