TL;DR
ID-Composer is a framework for multi-subject text-to-video generation that prevents subject identity drift and semantic confusion when synthesizing multiple specific people or objects. By combining a three-tiered hierarchical attention mechanism, a pretrained Vision-Language Model (VLM) for joint text-image reasoning, and online reinforcement learning via Flow-GRPO, it sets a new state of the art in subject fidelity and temporal coherence.
What Problem Does This Paper Address?
Current video foundation models excel at text-to-video and single-image animation, but generating scenes containing multiple specific subjects remains an open challenge. Existing approaches encounter three major failure modes:
- Identity degradation and cross-subject leakage: Subjects blend into one another or lose distinct features over time.
- Semantic conflict: Simply concatenating visual tokens of multiple subjects alongside text tokens increases sequence complexity and perplexity, causing the model to ignore textual constraints or confuse subject attributes.
- Temporal inconsistency: Two-stage pipelines (generating a keyframe first, then animating it) lack explicit subject correspondence across later frames and suffer from error accumulation.
Methods
Built upon the Wan-Video Rectified Flow latent diffusion transformer (DiT), ID-Composer introduces three core techniques:
- Hierarchical Identity-Preserving Attention: Instead of naive token concatenation, each transformer block computes attention across three progressive levels:
- Intra-subject attention: Processes tokens for each subject independently via self-attention to preserve fine-grained structural features.
- Inter-subject attention: Aggregates the resulting representations across all subjects to model inter-subject spatial relations without attribute mixing.
- Multi-modal attention: Fuses combined subject features with prompt tokens and latent video tokens to enforce cross-modal interaction.
- Semantic Reasoning via Pretrained VLM: Replaces conventional text-only encoders (such as CLIP or standard T5 alone) with Qwen2.5-VL-7B-Instruct combined with T5. The VLM accepts both text prompts and reference images simultaneously, generating joint context-aware conditioning tokens ($f_{txt}$) that describe complex multi-subject interactions before entering the DiT.
- Online Reinforcement Learning (Flow-GRPO): Standard flow-matching regression losses often fail to align high-level identity concepts. ID-Composer applies Flow-GRPO post-training by injecting stochasticity into the flow trajectory and optimizing a reward function balancing visual quality ($R_{qual}$) and reference identity consistency ($R_{id}$).
[Reference Images + Text]
│
├──> [Qwen2.5-VL / T5] ────> Contextual Text Tokens
└──> [Wan-VAE] ────> Multi-Subject Visual Tokens
│
┌───────────────────────┴───────────────────────┐
▼ ▼
[Intra-Subject Attention] │
│ │
▼ │
[Inter-Subject Attention] │
│ │
└───────────────> [Multi-Modal Attention] <─────┘
│
[Latent Video DiT]
│
[Flow-GRPO Reward Tuning]
Datasets
- Curated Training Dataset:
- OpenS2V-Nexus subset: 218,230 real-world video clips paired with 535,259 masked reference subject images.
- Synthetic Data: 9,374 high-fidelity video clips and ~3,000 subject images synthesized using in-context prompting via Gemini 2.5 Flash Image.
- Subject Image Prior Library: 14,976 static images (2,877 human-centric portraits and 3,458 isolated garments).
- Evaluation Benchmark: Evaluated on the OpenS2V-Nexus test suite (180 diverse multi-subject text pairs) using metrics including Face Similarity (FaceSim), text-video alignment (GmeScore), video-subject consistency (NexusScore), and perceptual realism (NaturalScore).
Key Insights and Ideas
- Decoupling attention stages prevents concept bleed: Allowing tokens from distinct subjects to cross-attend only after intra-subject self-attention isolates fine-grained visual features, preventing subject identity confusion (e.g., swapping clothing or facial attributes).
- VLMs outperform pure text encoders for complex conditioning: Multi-subject prompts require joint visual-textual reasoning (e.g., "the woman in image A hands the coffee mug from image B to the man in image C"). A multimodal LLM acts as an implicit scene planner, translating multimodal inputs into coherent conditioning tokens.
- Online RL effectively polishes identity preservation: Standard diffusion loss treats all latent dimensions equally, missing subtle identity drift. Flow-GRPO guided by an identity verification reward directly penalizes facial and feature drift across time steps.
Key Results
- Benchmark Superiority: On the OpenS2V-Nexus 180-pair benchmark, ID-Composer outperforms open-source and proprietary baselines:
- ID-Composer-14B achieves 57.05% Total Score and 60.50% FaceSim, outperforming Kling 1.6 (54.46% Total / 40.10% FaceSim), VACE-14B (52.87% Total / 55.09% FaceSim), and Phantom-14B (52.32% Total / 51.48% FaceSim).
- The base 1.3B model scores 54.33% Total with 58.12% FaceSim, rivaling models ten times its size.
- Ablation Studies:
- Removing hierarchical attention causes FaceSim to plunge from 58.12% to 51.34%.
- Removing the VLM encoder drops Text-Video Alignment from 49.55% to 42.17%.
- Omitting the curated synthetic/filtered data drops overall quality from 48.91% to 45.13%.
- Zero-Shot Video Editing: The architecture generalizes natively to video editing tasks, including subject replacement and scene compositing.
Limitations and Future Work
- Computational Demand: Stacking multi-level attention across large DiT backbones (up to 14B parameters) combined with a 7B VLM introduces significant inference and training latency.
- Complex Physical Dynamics: The framework occasionally struggles with intricate physical interactions and fine-grained motion trajectories between multiple subjects.
- Future Directions: The authors highlight developing lighter-weight architectures, incorporating 3D-aware physical priors, reducing foundational biases inherited from pretrained VLMs, and enhancing user control over fine-grained sub-actions.
Summary updated