Follow-Your-MultiPose: Tuning-Free Multi-Character Text-to-Video Generation via Pose Guidance

Beiyuan Zhang, Yue Ma, Chunlei Fu, Xinyang Song, Zhenan Sun, Ziqiang Li

Introduction: The Challenge of Multi-Character Synthesis

While diffusion-based video generation has made massive strides, most existing models struggle with "multi-character" scenarios. If you ask a standard model to generate "Ironman and Batman in a forest," the attributes often bleed together—a phenomenon known as "information contamination." For instance, Batman might end up with red armor because the global prompt applies to the entire frame.

Ironman and Batman generated with distinct identities using FYM's pose guidance.

Aligning multiple distinct pose sequences to specific characters is typically computationally expensive and often requires fine-tuning. Follow-Your-MultiPose (FYM) addresses this with a tuning-free framework that enables precise, multi-character text-to-video generation guided by poses. By decoupling complex prompts into individual character descriptions and using spatially aligned attention masks, it allows multiple distinct characters to interact in a single scene with high temporal and pose consistency.

Methodology

The FYM framework introduces four key modules that work on top of pre-trained models like Stable Diffusion and AnimateDiff:

  1. Mask Enhancement Flow: Spatial masks are extracted from pose sequences to act as "bounding boxes," defining exactly which pixels belong to which character.
  2. Prompt Filtering Module: An LLM (e.g., GPT-4) parses complex prompts into individual descriptions (e.g., "Ironman in a forest" and "Batman in a forest").
  3. Spatial Aligned Cross-Attention: The model performs parallel attention calculations for each character's specific prompt, merging them using spatial masks to ensure Ironman’s description only influences Ironman’s pixels.
  4. Multi-branch Control Module: Parallel ControlNet branches are used for each character, with features fused using enhanced masks before being fed into the U-Net.

Because the method is tuning-free, it can be plugged into various personalized checkpoints (like those from CivitAI) to generate videos in different artistic styles without retraining.

Key Results and Insights

By treating generation as parallel single-subject tasks stitched at the feature level, FYM achieves high temporal and pose consistency. The authors compared FYM against state-of-the-art methods using metrics like CLIP Score (CS) and Pose Accuracy (PA).

Multi-character generation in an anime style, demonstrating spatial separation and identity preservation.

FYM achieved a 32.79 CLIP Score and 35.32% Pose Accuracy, significantly outperforming baselines. In a user study, FYM was preferred by 82.8% of participants. Qualitatively, the model excels at maintaining distinct identities even in complex scenes with multiple characters—such as the Sailor Moon characters shown above—maintaining high-quality textures that baselines failed to match.

Limitations and Future Work

Despite its success, FYM faces challenges with computational overhead due to running multiple ControlNet branches and parallel attention passes. Additionally, extreme character occlusion can lead to limb separation issues, and the output quality remains dependent on the accuracy of the input pose extractor (e.g., DWPose). Future work may involve optimizing the multi-branch architecture to reduce computational costs.

Summary updated

BibTeX
@inproceedings{zhang2025follow,
  author = {Beiyuan Zhang and Yue Ma and Chunlei Fu and Xinyang Song and Zhenan Sun and Ziqiang Li},
  title = {{Follow-Your-MultiPose: Tuning-Free Multi-Character Text-to-Video Generation via Pose Guidance}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890233},
}
Figure