ID-Composer: Multi-Subject Video Synthesis with Hierarchical Identity Preservation

Panwang Pan, Jingjing Zhao, Yuchen Lin, Chenguo Lin, Chenxin Li, Honglei Yan, Tingting Shen, Yadong MuSubmission to ICLR 2026 · Withdrawn

TL;DR

ID-Composer is a framework for multi-subject text-to-video generation that prevents subject identity drift and semantic confusion when synthesizing multiple specific people or objects. By combining a three-tiered hierarchical attention mechanism, a pretrained Vision-Language Model (VLM) for joint text-image reasoning, and online reinforcement learning via Flow-GRPO, it sets a new state of the art in subject fidelity and temporal coherence.

What Problem Does This Paper Address?

Current video foundation models excel at text-to-video and single-image animation, but generating scenes containing multiple specific subjects remains an open challenge. Existing approaches encounter three major failure modes:

  1. Identity degradation and cross-subject leakage: Subjects blend into one another or lose distinct features over time.
  2. Semantic conflict: Simply concatenating visual tokens of multiple subjects alongside text tokens increases sequence complexity and perplexity, causing the model to ignore textual constraints or confuse subject attributes.
  3. Temporal inconsistency: Two-stage pipelines (generating a keyframe first, then animating it) lack explicit subject correspondence across later frames and suffer from error accumulation.

Methods

Built upon the Wan-Video Rectified Flow latent diffusion transformer (DiT), ID-Composer introduces three core techniques:

  • Hierarchical Identity-Preserving Attention: Instead of naive token concatenation, each transformer block computes attention across three progressive levels:
  • Intra-subject attention: Processes tokens for each subject independently via self-attention to preserve fine-grained structural features.
  • Inter-subject attention: Aggregates the resulting representations across all subjects to model inter-subject spatial relations without attribute mixing.
  • Multi-modal attention: Fuses combined subject features with prompt tokens and latent video tokens to enforce cross-modal interaction.
  • Semantic Reasoning via Pretrained VLM: Replaces conventional text-only encoders (such as CLIP or standard T5 alone) with Qwen2.5-VL-7B-Instruct combined with T5. The VLM accepts both text prompts and reference images simultaneously, generating joint context-aware conditioning tokens ($f_{txt}$) that describe complex multi-subject interactions before entering the DiT.
  • Online Reinforcement Learning (Flow-GRPO): Standard flow-matching regression losses often fail to align high-level identity concepts. ID-Composer applies Flow-GRPO post-training by injecting stochasticity into the flow trajectory and optimizing a reward function balancing visual quality ($R_{qual}$) and reference identity consistency ($R_{id}$).
[Reference Images + Text] 
         │
         ├──> [Qwen2.5-VL / T5] ────> Contextual Text Tokens
         └──> [Wan-VAE]         ────> Multi-Subject Visual Tokens
                                              │
                      ┌───────────────────────┴───────────────────────┐
                      ▼                                               ▼
         [Intra-Subject Attention]                                    │
                      │                                               │
                      ▼                                               │
         [Inter-Subject Attention]                                    │
                      │                                               │
                      └───────────────> [Multi-Modal Attention] <─────┘
                                                    │
                                            [Latent Video DiT]
                                                    │
                                      [Flow-GRPO Reward Tuning]

Datasets

  • Curated Training Dataset:
  • OpenS2V-Nexus subset: 218,230 real-world video clips paired with 535,259 masked reference subject images.
  • Synthetic Data: 9,374 high-fidelity video clips and ~3,000 subject images synthesized using in-context prompting via Gemini 2.5 Flash Image.
  • Subject Image Prior Library: 14,976 static images (2,877 human-centric portraits and 3,458 isolated garments).
  • Evaluation Benchmark: Evaluated on the OpenS2V-Nexus test suite (180 diverse multi-subject text pairs) using metrics including Face Similarity (FaceSim), text-video alignment (GmeScore), video-subject consistency (NexusScore), and perceptual realism (NaturalScore).

Key Insights and Ideas

  • Decoupling attention stages prevents concept bleed: Allowing tokens from distinct subjects to cross-attend only after intra-subject self-attention isolates fine-grained visual features, preventing subject identity confusion (e.g., swapping clothing or facial attributes).
  • VLMs outperform pure text encoders for complex conditioning: Multi-subject prompts require joint visual-textual reasoning (e.g., "the woman in image A hands the coffee mug from image B to the man in image C"). A multimodal LLM acts as an implicit scene planner, translating multimodal inputs into coherent conditioning tokens.
  • Online RL effectively polishes identity preservation: Standard diffusion loss treats all latent dimensions equally, missing subtle identity drift. Flow-GRPO guided by an identity verification reward directly penalizes facial and feature drift across time steps.

Key Results

  • Benchmark Superiority: On the OpenS2V-Nexus 180-pair benchmark, ID-Composer outperforms open-source and proprietary baselines:
  • ID-Composer-14B achieves 57.05% Total Score and 60.50% FaceSim, outperforming Kling 1.6 (54.46% Total / 40.10% FaceSim), VACE-14B (52.87% Total / 55.09% FaceSim), and Phantom-14B (52.32% Total / 51.48% FaceSim).
  • The base 1.3B model scores 54.33% Total with 58.12% FaceSim, rivaling models ten times its size.
  • Ablation Studies:
  • Removing hierarchical attention causes FaceSim to plunge from 58.12% to 51.34%.
  • Removing the VLM encoder drops Text-Video Alignment from 49.55% to 42.17%.
  • Omitting the curated synthetic/filtered data drops overall quality from 48.91% to 45.13%.
  • Zero-Shot Video Editing: The architecture generalizes natively to video editing tasks, including subject replacement and scene compositing.

Limitations and Future Work

  • Computational Demand: Stacking multi-level attention across large DiT backbones (up to 14B parameters) combined with a 7B VLM introduces significant inference and training latency.
  • Complex Physical Dynamics: The framework occasionally struggles with intricate physical interactions and fine-grained motion trajectories between multiple subjects.
  • Future Directions: The authors highlight developing lighter-weight architectures, incorporating 3D-aware physical priors, reducing foundational biases inherited from pretrained VLMs, and enhancing user control over fine-grained sub-actions.

Summary updated

Abstract

Video generative models pretrained on large-scale datasets can produce high-quality videos, but are often conditioned on text or a single image, limiting controllability and applicability. We introduce ID-Composer, a novel framework that addresses this gap by tackling multi-subject video generation from a text prompt and reference images. This task is challenging as it requires preserving subject identities, integrating semantics across subjects and modalities, and maintaining temporal consistency. To faithfully preserve the subject consistency and textual information in synthesized videos, ID-Composer~designs a **hierarchical identity-preserving attention mechanism**, which effectively aggregates features within and across subjects and modalities. To effectively allow for the semantic following of user intention, we introduce **semantic understanding via pretrained vision-language model (VLM)**, leveraging VLM's superior semantic understanding to provide fine-grained guidance and capture complex interactions between multiple subjects. Considering that standard diffusion loss often fails in aligning the critical concepts like subject ID, we employ an **online reinforcement learning phase** to drive the overall training objective of ID-Composer into RLVR. Extensive experiments demonstrate that our model surpasses existing methods in identity preservation, temporal consistency, and video quality. Code and training data will be released.
BibTeX
@inproceedings{pan2026id,
  author = {Panwang Pan and Jingjing Zhao and Yuchen Lin and Chenguo Lin and Chenxin Li and Honglei Yan and Tingting Shen and Yadong MU},
  title = {{ID-Composer: Multi-Subject Video Synthesis with Hierarchical Identity Preservation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=8ski8zjiK1},
}
Figure