ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation

Panwang Pan, Jingjing Zhao, Yuchen Lin, Chia‐Liang Lin, Chenxin Li, Haopeng Li, Honglei Yan, Tsu‐Wang Shen

Abstract

Significant progress has been achieved in high-fidelity video synthesis, yet current paradigms often fall short in effectively integrating identity information from multiple subjects. This leads to semantic conflicts and suboptimal performance in preserving identities and interactions, limiting controllability and applicability. To tackle this issue, we introduce ID-Crafter, a framework for multi-subject video generation that achieves superior identity preservation and semantic coherence. ID-Crafter integrates three key components: (i) a hierarchical identity-preserving attention mechanism that progressively aggregates features at intra-subject, inter-subject, and cross-modal levels; (ii) a semantic understanding module powered by a pretrained Vision-Language Model (VLM) to provide fine-grained guidance and capture complex inter-subject relationships; and (iii) an online reinforcement learning phase to further refine the model for critical concepts. Furthermore, we construct a new dataset to facilitate robust training and evaluation. Extensive experiments demonstrate that ID-Crafter establishes new state-of-the-art performance on multi-subject video generation benchmarks, excelling in identity preservation, temporal consistency, and overall video quality.

Introduction

While text-to-video models have reached impressive levels of realism, generating videos with multiple specific subjects remains a significant challenge. Most existing models suffer from "identity leakage"—where features of one subject bleed into another—and "semantic conflicts," where the model fails to correctly assign actions to the intended subjects. ID-Crafter addresses these issues by moving beyond simple feature concatenation, instead employing a structured approach to identity preservation and semantic grounding.

ID-Crafter demonstrates the ability to compose multiple distinct elements—a specific face, a garment, and an object—into a coherent, high-fidelity video scene.

Methodology

ID-Crafter is built upon a latent diffusion transformer (DiT) backbone and introduces three primary innovations:

1. Hierarchical Identity-Preserving Attention

To prevent identity bleed, ID-Crafter processes subject tokens through a three-stage attention hierarchy:

  • Intra-subject attention: Refines individual subject features in isolation to capture fine-grained details.
  • Gated Inter-subject attention: Models the relationships and interactions between subjects while strictly maintaining identity boundaries.
  • Cross-modal attention: Fuses these refined subject features with text and video tokens to ensure the final output aligns with the global prompt.

2. VLM-Based Semantic Grounding

Traditional text encoders like CLIP often lack the depth required for complex, multi-subject prompts. ID-Crafter utilizes a pretrained Vision-Language Model (Qwen2.5-VL) as a dual text-image encoder. This allows the model to perform granular, cross-modal reasoning, effectively acting as a semantic router that guides the spatial and temporal layout of the generated video.

3. Online Reinforcement Learning (Flow-GRPO)

To optimize the balance between identity fidelity and aesthetic quality, the framework employs an online RL post-training stage. Using a novel algorithm called Flow-GRPO, the model is fine-tuned based on a composite reward function that evaluates face similarity, subject consistency, and overall physical plausibility. By utilizing groupwise comparisons rather than a value network, Flow-GRPO provides a stable and effective way to refine video diffusion models.

Data and Training

The authors curated a massive dataset to support complex multi-subject training, including the OpenS2V-Nexus collection (218k videos) and a high-quality internal collection of 9.3k professionally shot videos. To avoid the artifacts of simple data augmentation, they used an advanced VLM pipeline to synthesize coherent multi-subject reference images, providing the model with rich priors for appearance and interaction.

The framework excels at preserving the identity of well-known subjects while placing them in dynamic, complex environments with high temporal consistency.

Key Results and Evaluation

ID-Crafter sets a new state-of-the-art for multi-subject video generation:

  • Benchmark Performance: The 14B parameter version achieved a total score of 57.05% on the OpenS2V-Nexus benchmark, outperforming both open-source (Phantom, SkyReels) and proprietary (Kling 1.6, Vidu) models.
  • Impact of RL: The online RL stage significantly boosted performance, yielding a 13.7% increase in face similarity and a 14.9% increase in aesthetic scores over the supervised fine-tuning baseline.
  • Human Preference: In user studies, ID-Crafter was preferred 60% of the time for identity consistency and 65% of the time for motion naturalness compared to leading competitors.

Conclusion

By combining hierarchical attention, VLM-driven semantic grounding, and stable online reinforcement learning, ID-Crafter effectively solves the core problems of identity leakage and semantic confusion in multi-subject video generation. While challenges remain in modeling extremely complex physical interactions, ID-Crafter represents a major step forward in controllable and compositional video synthesis.

Summary updated

Abstract

Significant progress has been achieved in high-fidelity video synthesis, yet current paradigms often fall short in effectively integrating identity information from multiple subjects. This leads to semantic conflicts and suboptimal performance in preserving identities and interactions, limiting controllability and applicability. To tackle this issue, we introduce ID-Crafter, a framework for multi-subject video generation that achieves superior identity preservation and semantic coherence. ID-Crafter integrates three key components: (i) a hierarchical identity-preserving attention mechanism that progressively aggregates features at intra-subject, inter-subject, and cross-modal levels; (ii) a semantic understanding module powered by a pretrained Vision-Language Model (VLM) to provide fine-grained guidance and capture complex inter-subject relationships; and (iii) an online reinforcement learning phase to further refine the model for critical concepts. Furthermore, we construct a new dataset to facilitate robust training and evaluation. Extensive experiments demonstrate that ID-Crafter establishes new state-of-the-art performance on multi-subject video generation benchmarks, excelling in identity preservation, temporal consistency, and overall video quality. Project page: https://angericky.github.io/ID-Crafter
BibTeX
@inproceedings{pan2026id,
  author = {Panwang Pan and Jingjing Zhao and Yuchen Lin and Chia‐Liang Lin and Chenxin Li and Haopeng Li and Honglei Yan and Tsu‐Wang Shen},
  title = {{ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=8ski8zjiK1},
}
Figure