Abstract
Human image animation—generating video sequences from a single reference image and a target pose sequence—often suffers from detail blurring, identity drift, and background artifacts. We present ReMask-Animate, a novel framework that utilizes mask-guided adapters to address these challenges. By employing region-specific masks to direct the diffusion model's attention, our approach effectively decouples foreground and background processing while fusing multiple pose modalities. Experimental results demonstrate that ReMask-Animate achieves state-of-the-art performance in temporal consistency and visual fidelity, particularly in preserving intricate details like hands and facial identity.
Introduction
The ability to animate a static character image based on a sequence of poses is a significant challenge in digital content creation. While recent diffusion-based models have shown promise, they frequently encounter three primary issues: feature confusion between the subject and background, loss of fine-grained details in complex regions like hands and faces, and identity drift over time. These problems typically arise from the model's inability to precisely allocate its attention to relevant spatial regions during the generation process.
Methodology
ReMask-Animate introduces three lightweight, mask-guided adapters integrated into a diffusion backbone (U-Net with a ReferenceNet) to provide fine-grained control over the generation process.
Mask-Guided Adapters
- Mask-guided P-Adapter (Pose): This module fuses sparse pose keypoints (e.g., DWPose) with dense pose representations (e.g., SMPL). A gated unit and specific masks for hands and faces dynamically weight these signals, ensuring precise control over critical body parts while maintaining overall volumetric consistency.
- Mask-guided S-Adapter (Spatial/Layout): Located within the U-Net decoder, this adapter uses foreground and background masks to separate feature calculations. This prevents "feature bleeding," ensuring the background remains stable while the character moves.
- Mask-guided C-Adapter (Character/Identity): To maintain identity, high-fidelity facial features are extracted using FaceNet and CLIP. These features are injected specifically into the foreground region defined by the mask, preserving the subject's likeness even during complex rotations.
Self-Evolution Mask Process
To handle inference where ground-truth masks are unavailable, the framework employs a recursive "self-evolution" process. Starting from a simple bounding box, a pre-trained segmentation network iteratively refines the mask over multiple cycles as the video frames are generated.
Experimental Results
The model was evaluated on the TikTok and Fashion datasets, which offer a range of movement complexity and background diversity.
Quantitative Performance
On the TikTok dataset, ReMask-Animate achieved a Structural Similarity Index (SSIM) of 0.785, a significant improvement over previous state-of-the-art models like MagicAnimate and AnimateAnyone. It also recorded a substantially lower FID-VID of 19.51, indicating superior video quality and temporal stability.
Qualitative Analysis
As shown in the comparative results, the framework produces remarkably crisp details. Hands maintain anatomical correctness without the "spaghetti" artifacts common in other methods. Faces remain consistent with the reference image even through 180-degree turns, and backgrounds remain sharp and static throughout the animation.
Discussion and Conclusion
The success of ReMask-Animate highlights the effectiveness of using masks as "attention traffic controllers" within neural networks. By forcing the model to focus on specific features only within their corresponding spatial regions, the framework significantly mitigates common animation artifacts. Furthermore, the synergy between sparse and dense pose modalities provides a more robust control signal than either used in isolation.
While the current iterative mask refinement process may impact real-time performance, the overall framework adds minimal computational overhead (approximately 27 million parameters). Future work will focus on optimizing inference speed and testing the model's robustness against extreme occlusions in large-scale, "in-the-wild" datasets.
Summary updated