Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints

Thiago Gomes, Renato Martins, Joao Ferreira, Erickson NascimentoWACV 2020

TL;DR

This paper introduces a novel framework for transferring human motion and appearance between monocular videos. It addresses the limitations of existing methods by jointly considering body shape, motion, and appearance, enabling more realistic and consistent video synthesis of a target character performing actions from a source video.

What Problem Does This Paper Address?

Creating realistic virtual humans that can mimic the movements and appearances of real actors from videos is a significant challenge in computer vision and graphics. While recent advances in view synthesis and image-to-image translation have shown promise, they often struggle to transfer not just visual style but also the intricate interplay of a character's body shape and motion. Existing methods tend to either stretch/squeeze textures to fit a new pose or fail to preserve crucial motion details like feet on the ground or hands interacting with objects. This paper aims to bridge this gap by developing a method that can transfer both motion and appearance from one monocular video to another, while respecting the target character's unique morphology and maintaining critical physical constraints.

Methods

The proposed framework is divided into four main components:

  1. Human Motion Estimation: The method first estimates the motion of the character in the source video. It builds upon the SMPL pose estimation framework but introduces a motion regularization step. This regularization uses an energy function that minimizes pose projection errors and encourages temporal motion smoothness by penalizing high-frequency artifacts, ensuring a consistent character motion.
  2. Target 3D Human Body Model Building: For the target character, a 3D body model is reconstructed. The paper leverages existing methods for initial 3D body model estimation (e.g., from [5]) and then employs a Generative Adversarial Network (GAN) for face texture restoration. This GAN is trained directly on texture maps to improve visual details and mitigate warping/interpolation artifacts, enhancing the realism of the target's appearance.
  3. Motion Retargeting with Space-time Constraints: This is the core of the novel approach. The estimated motion from the source video is adapted to the target character's body shape. Crucially, this process incorporates:
  4. Space-time Constraints: To preserve the style and naturalness of the original motion, a prediction error in 3D space is minimized over a temporal window.
  5. Spatial Motion Restrictions and Physical Interactions: Key physical constraints, such as feet on the floor or hands interacting with objects, are explicitly enforced by constraining end-effector positions.
  6. Space-time Cost Error Optimization: A final optimization combines motion appearance, shape differences, and physical restrictions to produce the retargeted motion.
  7. Image Rendering and Compositing: The retargeted motion is rendered onto the target character's 3D model. This rendered character is then composited into the background of the source video. The process involves segmenting the source background, inpainting any missing areas for temporal smoothness, and using Poisson blending for illumination adjustment to seamlessly integrate the synthesized character.

Datasets

  • Training: The paper mentions using videos from the People-Snapshot dataset [5] and videos of actors captured with a Point Grey camera for building target person shape and appearance. For GAN face texture restoration, they used 500 human face textures from 3D.sk¹ and augmented this dataset with noise, rotations, and warping.
  • Evaluation: The evaluation is performed on publicly available video sequences: "alfonso-ribeiro," "joao-pedro," "tom-cruise," and "bruno-mars." These sequences are chosen to represent challenging conditions like varying illumination, fast motions, and different actor morphologies.

Key Insights and Ideas

  • Jointly addressing motion and appearance: The paper highlights that simply transferring visual style (appearance) is insufficient for realistic human motion transfer. The interplay between body shape, motion, and appearance must be considered holistically.
  • Importance of physical constraints: Explicitly enforcing physical constraints (e.g., feet on the ground) is crucial for generating plausible motion, especially when transferring between characters with different morphologies. Naive transfer without these constraints leads to unnatural movements.
  • GANs for texture enhancement: Using GANs for texture denoising, particularly on face textures, significantly improves the visual quality and detail of the synthesized character.
  • Temporal consistency for natural motion: Regularizing motion estimation and using space-time constraints during retargeting are key to producing smooth and natural-looking movements, avoiding jittery or artifact-ridden results.
  • Dataset-specific vs. generalizable approach: The proposed method aims to be more generalizable than previous state-of-the-art approaches that often require training a separate GAN for each video, making it computationally intensive and less flexible.

Key Results

  • Quantitative Metrics: The proposed method outperforms the V-Unet baseline in terms of Structural Similarity (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) for appearance quality. It also achieves a lower number of missed 2D joint detections. For example, in the "alfonso-ribeiro" sequence, SSIM is 0.837 vs. 0.834 and LPIPS is 0.126 vs. 0.137.
  • Qualitative Results: The method produces visually richer and more detailed appearances of the target character compared to the baseline. It successfully preserves specific motion features, such as feet placement, and handles interactions with objects (e.g., hands on cones). Figure 8 showcases these improvements across various challenging sequences.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the current method primarily focuses on transferring motion and appearance.
  • While the method handles different body shapes, it assumes a homeomorphic skeleton structure (i.e., similar limb proportions, though not necessarily the same lengths).
  • The face texture restoration is limited to face textures; extending this to the entire body's texture is a potential area for improvement.
  • Future Work:
    • Automatically detecting retargeting motion constraints in videos.
  • Improving appearance restoration and transferring beyond just face texture maps.
  • Enhancing scene compositing, such as estimating scene illumination for more realistic rendering.

Summary updated

BibTeX
@inproceedings{gomes2020do,
  author = {Thiago Gomes and Renato Martins and Joao Ferreira and Erickson Nascimento},
  title = {{Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure