TL;DR
Follow-Your-MultiPose (FYM) is a tuning-free framework that enables precise, multi-character text-to-video generation guided by poses. By decoupling complex prompts into individual character descriptions and using spatially aligned attention masks, it prevents "information contamination," allowing multiple distinct characters to interact in a single scene with high temporal and pose consistency.
What Problem Does This Paper Address?
While diffusion-based video generation has made massive strides, most existing models struggle with "multi-character" scenarios. If you ask a standard model to generate "Ironman and Batman in a forest," the attributes often bleed together (e.g., Batman might end up with red armor).
This "information contamination" happens because standard cross-attention mechanisms apply a single global prompt to the entire image frame. Furthermore, aligning multiple distinct pose sequences to specific characters in a video is computationally expensive and often requires fine-tuning on specific datasets. The authors sought a tuning-free way to give users fine-grained control over who is in the video, where they are, and what they are doing.
Methods
The FYM framework introduces four key modules that work on top of pre-trained models like Stable Diffusion and AnimateDiff:
- Mask Enhancement Flow: Instead of just using raw pose skeletons, the system extracts spatial masks from the pose sequences. These masks act as "bounding boxes" that tell the model exactly which pixels belong to which character.
- Prompt Filtering Module: The system uses a Large Language Model (LLM) like GPT-4 or Llama to parse a complex multi-character prompt into separate, individual prompts (e.g., "Ironman in a forest" and "Batman in a forest").
- Spatial Aligned Cross-Attention: This is the core innovation. Instead of one global attention pass, the model performs multiple parallel attention calculations—one for each character's specific prompt. These are then merged using the spatial masks, ensuring Ironman’s description only influences Ironman’s pixels.
- Multi-branch Control Module: To ensure the poses are followed accurately without interference, the framework runs multiple parallel ControlNet branches (one per character). The features from these branches are then fused using the enhanced masks before being fed into the U-Net.
Because the method is tuning-free, it can be plugged into various personalized "Checkpoints" (like those from CivitAI) to generate videos in different artistic styles (Realistic, Anime, etc.) without any retraining.
Datasets
The paper does not introduce a new training dataset because the method is tuning-free. For evaluation and implementation:
- Base Models: Stable Diffusion V1.5 and AnimateDiff.
- Pose Extraction: DWPose was used to extract pose sequences from source videos to serve as guidance.
- Personalized Models: The authors tested the framework using various community checkpoints including Realistic Vision, Toon You, Anything, Raemumix, and HumanRealistic.
Key Insights and Ideas
- Parallelism over Fusion: Rather than trying to force a model to understand a complex multi-subject prompt all at once, FYM treats the generation as several parallel single-subject tasks that are spatially stitched together at the feature level.
- LLMs as Scene Parsers: Using LLMs to "clean" and "split" prompts ensures that the diffusion model receives precise instructions for each masked region, reducing semantic ambiguity.
- Masking the "Control": The authors found that simply adding poses isn't enough; you must mask the ControlNet features themselves. Without this, the "signal" from one character's pose can leak into the background or onto another character, causing visual artifacts.
Key Results
The authors compared FYM against several state-of-the-art methods (Follow Your Pose, ControlVideo, Tune-a-Video, etc.) using four metrics: CLIP Score (CS), Frame Consistency (FC), Pose Accuracy (PA), and a User Study (US).
- Superior Accuracy: FYM achieved the highest scores across the board, notably reaching a 32.79 CLIP Score and 35.32% Pose Accuracy, outperforming the next best models significantly.
- User Preference: In the user study, FYM was preferred by 82.8% of participants, highlighting its ability to maintain distinct character identities.
- Qualitative Excellence: The model successfully generated complex scenes—such as four distinct "Sailor Moon" characters on a beach or political figures in a park—maintaining high-quality textures and temporal stability that baselines failed to match.
Limitations and Future Work
- Computational Overhead: Running multiple ControlNet branches and parallel attention passes increases the VRAM requirements and inference time compared to single-character generation.
- Occlusion Challenges: While the masking helps, the paper notes that extreme overlapping of characters (where masks heavily intersect) can still be a challenge for maintaining perfect limb separation.
- Dependency on Pose Quality: The final output is only as good as the input pose sequence; if the pose extractor (DWPose) fails or produces jittery skeletons, the generated video will inherit those flaws.
Future work may involve optimizing the multi-branch architecture to reduce computational costs or integrating more complex interaction logic between characters.
Summary updated