Introduction: The Challenge of Multi-Character Synthesis
While diffusion-based video generation has made massive strides, most existing models struggle with "multi-character" scenarios. If you ask a standard model to generate "Ironman and Batman in a forest," the attributes often bleed together—a phenomenon known as "information contamination." For instance, Batman might end up with red armor because the global prompt applies to the entire frame.
Aligning multiple distinct pose sequences to specific characters is typically computationally expensive and often requires fine-tuning. Follow-Your-MultiPose (FYM) addresses this with a tuning-free framework that enables precise, multi-character text-to-video generation guided by poses. By decoupling complex prompts into individual character descriptions and using spatially aligned attention masks, it allows multiple distinct characters to interact in a single scene with high temporal and pose consistency.
Methodology
The FYM framework introduces four key modules that work on top of pre-trained models like Stable Diffusion and AnimateDiff:
- Mask Enhancement Flow: Spatial masks are extracted from pose sequences to act as "bounding boxes," defining exactly which pixels belong to which character.
- Prompt Filtering Module: An LLM (e.g., GPT-4) parses complex prompts into individual descriptions (e.g., "Ironman in a forest" and "Batman in a forest").
- Spatial Aligned Cross-Attention: The model performs parallel attention calculations for each character's specific prompt, merging them using spatial masks to ensure Ironman’s description only influences Ironman’s pixels.
- Multi-branch Control Module: Parallel ControlNet branches are used for each character, with features fused using enhanced masks before being fed into the U-Net.
Because the method is tuning-free, it can be plugged into various personalized checkpoints (like those from CivitAI) to generate videos in different artistic styles without retraining.
Key Results and Insights
By treating generation as parallel single-subject tasks stitched at the feature level, FYM achieves high temporal and pose consistency. The authors compared FYM against state-of-the-art methods using metrics like CLIP Score (CS) and Pose Accuracy (PA).
FYM achieved a 32.79 CLIP Score and 35.32% Pose Accuracy, significantly outperforming baselines. In a user study, FYM was preferred by 82.8% of participants. Qualitatively, the model excels at maintaining distinct identities even in complex scenes with multiple characters—such as the Sailor Moon characters shown above—maintaining high-quality textures that baselines failed to match.
Limitations and Future Work
Despite its success, FYM faces challenges with computational overhead due to running multiple ControlNet branches and parallel attention passes. Additionally, extreme character occlusion can lead to limb separation issues, and the output quality remains dependent on the accuracy of the input pose extractor (e.g., DWPose). Future work may involve optimizing the multi-branch architecture to reduce computational costs.
Summary updated