Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation

Zhenghao Zhang, Junchao Liao, Xiangyu Meng, Long Qin, Weizhi Wang

Introduction

The evolution of Diffusion Transformers (DiT) has significantly advanced the field of video generation. However, achieving precise control over both the visual identity of multiple entities and their specific motion trajectories remains a formidable challenge. Tora2 emerges as a state-of-the-art framework designed to bridge this gap, offering a tuning-free solution for the simultaneous customization of appearance and motion. By integrating reference images and hand-drawn trajectories, Tora2 enables the generation of high-fidelity videos where distinct characters or objects adhere to user-defined paths while maintaining their unique visual characteristics.

Problem Statement: The Challenges of Fine-Grained Control

Existing large-scale video generation models often struggle with two primary issues when tasked with complex, multi-subject scenes:

  1. Multimodal Conflict: There is a persistent difficulty in reconciling appearance constraints (e.g., a specific face) with motion constraints (e.g., a specific trajectory). Models often prioritize one at the expense of the other.
  2. Multi-Entity Confusion: In scenes featuring multiple subjects, such as a person and a pet, models frequently suffer from "identity bleeding" or fail to correctly assign specific motions to the intended subjects.

Traditional methods often rely on time-consuming and computationally expensive fine-tuning. Tora2 addresses these hurdles by providing a robust architecture that handles multi-subject scenarios with high physical and visual fidelity without requiring per-case tuning.

Methodology

Tora2 builds upon the trajectory-focused foundations of the original Tora model, introducing several architectural innovations to handle appearance and multi-entity binding.

Decoupled Personalization Extractor (DPE)

Standard global semantic features (e.g., DINOv2) often lack the granularity required for high-fidelity identity preservation, particularly for human faces. The DPE addresses this by utilizing specialized encoders—ArcFace for facial features and ReID for objects. These high-frequency details are fused with global features using a Q-Former, ensuring that the generated entities remain faithful to the reference images.

Gated Self-Attention and Binding Strategy

To manage multiple entities within a single scene, Tora2 employs a gated self-attention mechanism. This "binding" strategy explicitly links specific visual entities to their corresponding text descriptions and motion trajectories. This mechanism acts as a spatial traffic controller, preventing attributes from one entity from leaking into another and ensuring that each subject follows its designated path.

Hierarchical Injection and Loss Functions

The timing of condition injection is critical. Tora2 utilizes hierarchical injection, where motion embeddings are infused early via adaptive layer normalization to modulate latents, while appearance embeddings are introduced later through cross-attention layers. Research indicated that injecting motion before appearance prevents the "dampening" of motion signals, which otherwise results in static characters. Furthermore, a Dual-Branch Contrastive Loss is employed during training to strengthen the mathematical correspondence between visual features and motion patterns.

Data and Evaluation

The model was trained on a massive, curated dataset of 1.1 million video clips. This dataset was constructed by merging pipelines from Video Alchemist and Tora, utilizing automated tools like LISA for segmentation and CoTracker3 for tracking to create precise entity-text-trajectory triplets.

Evaluation was conducted using the MSRVTT-Personalization benchmark (2,130 clips) and a custom "multi-concept" set of 200 videos. These benchmarks tested the model's ability to handle complex, multi-subject scenarios in diverse environments.

Key Results and Insights

Tora2 demonstrates significant improvements over two-stage baselines, such as combining Flux.1 for image generation with Tora for video.

  • Trajectory Accuracy: On the MSRVTT benchmark, Tora2 achieved a trajectory error of 13.52 for human entities, a notable improvement over the baseline's 17.41.
  • Identity Preservation: The model achieved a Face Similarity (Face-S) score of 0.419, outperforming Video Alchemist (0.411) and the Tora+Flux baseline (0.363).
  • Qualitative Performance: Qualitative results highlight Tora2's ability to process complex prompts—such as a specific man and a specific dog walking on the moon—while maintaining distinct identities and trajectories for each.

The research underscores that high-frequency features are essential for human-centric video, as the human eye is highly sensitive to facial inconsistencies. By decoupling these features and employing a strategic injection order, Tora2 maintains superior identity permanence and motion fluidity.

Limitations and Future Work

Despite its successes, Tora2 is not without limitations. Its performance is heavily dependent on the quality of the automated masks and tracklets used during training; errors in the data pipeline can lead to generative artifacts. Additionally, while the model excels at independent trajectories, complex physical interactions—such as two people hugging—remain a challenge for current diffusion-based architectures. Future research may focus on integrating more granular controls, such as pose-based guidance, to enhance the expressiveness of character animations.

Summary updated

BibTeX
@inproceedings{zhang2025tora,
  author = {Zhenghao Zhang and Junchao Liao and Xiangyu Meng and Long Qin and Weizhi Wang},
  title = {{Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3754869},
}
Figure