Abstract
Video-based human motion transfer creates video animations of humans following a source motion. Current methods show remarkable results for tightly-clad subjects. However, the lack of temporally consistent handling of plausible clothing dynamics, including fine and high-frequency details, significantly limits the attainable visual quality. We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments. In contrast to the previous techniques, we perform image generation in three subsequent stages, synthesizing human shape, structure, and appearance. Given a monocular RGB video of an actor, we train a stack of recurrent deep neural networks that generate these intermediate representations from 2D poses and their temporal derivatives. Splitting the difficult motion transfer problem into subtasks that are aware of the temporal motion context helps us to synthesize results with plausible dynamics and pose-dependent detail. It also allows artistic control of results by manipulation of individual framework stages. In the experimental results, we significantly outperform the state-of-the-art in terms of video realism.
Introduction
Human motion transfer, often referred to as "performance cloning," aims to animate a target person in a video based on the movements of a source actor. While deep learning has advanced this field, existing models often suffer from a "tight-clothing bias." They perform well on form-fitting attire but struggle with loose garments like t-shirts, hoodies, or dresses. The primary challenge lies in the complex, non-rigid dynamics of loose fabric—swings, folds, and shadows—which do not follow a simple 1-to-1 relationship with the underlying skeleton. Standard pose-to-image networks frequently produce jittery textures or artifacts because they lack an understanding of fabric physics and temporal consistency.
This research introduces a multi-stage neural framework that explicitly models 2D shape and internal clothing structure (wrinkles and folds) in a recurrent fashion. By decomposing the problem, the system achieves photorealistic results that handle loose garments and complex fabric dynamics far better than previous state-of-the-art methods.
Proposed Method
Rather than employing a single "black box" network, the authors propose a cascaded architecture consisting of four specialized generative networks. This decomposition makes the complex mapping from pose to pixels more manageable and better-conditioned.
Multi-Stage Architecture
- Shape Estimator ($G_{shp}$): Predicts a semantic segmentation mask (silhouette) of the person, identifying limbs and garment types.
- Structure Estimator ($G_{str}$): A key innovation that estimates the internal gradient structure of clothing—a map of wrinkles and folds. It utilizes oriented Gabor filters to learn these patterns.
- Appearance Network ($G_{app}$): Synthesizes the actual pixels of the actor based on the predicted shape and wrinkle structure.
- Refinement Network ($G_{ref}$): Blends the synthesized actor into a static background, smoothing edges and adding realistic shadows.
To ensure temporal consistency and realistic dynamics, the networks are recurrent, conditioning each frame's prediction on the previous output. Furthermore, the input includes the first and second derivatives of the pose (velocity and acceleration), allowing the model to approximate physical momentum, such as the swing of a skirt.
Intermediate Representations
The use of intermediate representations is central to the framework's success. By treating fabric wrinkles as an explicit "Structure" map, the model maintains high-frequency details that are typically lost in standard GAN-based approaches.
This decoupling also enables artistic control, such as "Material Editing," where the shape and wrinkle structure from one source can be combined with the appearance (color and texture) of a different target actor.
Experimental Results
The researchers introduced a new high-fidelity dataset featuring subjects in various loose clothing styles to address the limitations of existing benchmarks. Quantitative evaluations showed that the proposed method outperformed state-of-the-art models like Everybody Dance Now (EDN) and pix2pixHD across metrics such as SSIM (Structural Similarity) and LPIPS (Perceptual Distance).
Qualitatively, user studies confirmed that the results were significantly more realistic, particularly in maintaining natural motion and garment deformation. Despite the multi-stage complexity, the system remains efficient, processing frames in approximately 280ms on a single GPU.
Conclusion and Future Work
This framework demonstrates that decomposing human motion transfer into shape, structure, and appearance stages significantly improves the handling of loose clothing. However, limitations remain, including a dependence on the variety of poses in training data and the accuracy of 2D pose detection and initial segmentation. Future work may involve incorporating 3D constraints and training on unpaired data to further enhance the range and realism of synthesized motions.
Summary updated