Abstract
Human Video Motion Transfer (HVMT) aims to animate a source person based on the movements of a driving video. Traditional methods relying on flow-based warping often suffer from "melting" artifacts and blurry textures when faced with complex poses or scale differences. We present REMOT (REgion-to-whole human MOtion Transfer), a framework that abandons global warping in favor of a progressive, part-based generation strategy. By synthesizing body parts individually and compositing them into a whole, REMOT achieves state-of-the-art realism and temporal consistency across challenging benchmarks.
Introduction
Human Video Motion Transfer is the "digital puppetry" task of taking a single image of a person (the source) and making them move according to a video of someone else (the driving video). The industry standard has long relied on Generative Adversarial Networks (GANs) combined with flow-based warping, where the model attempts to calculate how to "stretch" pixels from the source image to fit a new pose.
However, this approach often fails spectacularly when there are dramatic differences in scale (e.g., a tall person driving a short person) or complex poses (e.g., limbs crossing). The result is often ghosting limbs and distorted textures. REMOT addresses these issues by moving away from global warping toward a hierarchical generation process.
The Region-to-Whole Framework
REMOT breaks the generation process into three distinct stages to ensure structural integrity and texture preservation:
- Layout Generation: Instead of generating pixels directly, the model first predicts a semantic layout (parsing masks for the head, torso, limbs, etc.) based on the driving pose. This provides a structural blueprint for the final image.
- Region Generation: A specialized Region GAN generates each body part individually. To ensure the source person's features are correctly sized, the authors introduced a Global Alignment Module (GAM). This module uses affine transformations to match the scale and position of the source person to the driving person's layout before features are fused.
- Whole Composition: Finally, the Whole Composition Network (WCN) assembles these parts and adds the background. To maintain sharp clothing textures, a Texture Alignment Module (TAM) uses a similarity-based attention mechanism to "find" the right textures from the source image, rather than geometrically warping them.
Key Insights
- Abandoning the Flow: Explicit flow-based warping is a bottleneck. By treating motion transfer as a progressive generation task (Layout -> Parts -> Whole), the model avoids the mathematical errors inherent in calculating optical flow between disparate human shapes.
- Semantic Consistency: Using human parsing (segmenting the body into functional parts) rather than just skeletal keypoints allows the model to maintain better "object permanence" for clothing and limbs.
- Texture as a Search Problem: The TAM treats texture preservation as a similarity matching problem, which is more robust than 2D warping when handling occlusions and rotations.
Evaluation and Results
REMOT was evaluated on two major benchmarks: the iPER Dataset (controlled lab environment) and the SoloDance Dataset (complex real-world dance moves).
REMOT achieved state-of-the-art (SOTA) performance across several metrics. On the iPER dataset, it outperformed existing methods like Liquid Warping GAN (LWGAN) in SSIM (0.856), PSNR (25.33), and LPIPS. Crucially, it scored significantly higher on the Temporal Consistency Metric (TCM), indicating smoother videos with less flickering. Qualitative results show much clearer faces and sharper clothing textures, maintaining coherent human structure even when the driving person performs complex rotations.
Conclusion and Future Work
REMOT demonstrates that a region-based generative approach is superior to global warping for realistic human motion transfer. While the model currently operates in 2D and requires significant data to generalize to unseen environments, future work may incorporate 3D human body information (such as SMPL models) to provide even more powerful structural priors for handling extreme occlusions.
Summary updated