MoRF: Mobile Realistic Fullbody Avatars From a Monocular Video

Renat Bashirov, Alexey Larionov, Evgeniya Ustinova, Mikhail Sidorenko, David Svitov, Ilya Zakharkin, Victor LempitskyWACV 2024

TL;DR

This paper introduces MoRF, a system that generates highly realistic, full-body human avatars from short monocular videos, capable of real-time rendering on mobile devices. MoRF achieves this by introducing a novel "neural texture warping" technique that improves alignment and temporal consistency, overcoming limitations of prior methods.

What Problem Does This Paper Address?

Creating realistic, full-body human avatars from video has been a significant challenge. Existing methods often struggle with one or more of the following: requiring multi-view or depth sensors, being too slow for real-time mobile rendering, taking prohibitively long to train, or producing avatars with low image sharpness and temporal inconsistencies. Specifically, while methods like Deferred Neural Rendering (DNR) offer fast rendering, they can be sensitive to geometric inaccuracies and frame-to-frame misalignments, leading to artifacts. This paper aims to bridge this gap by enabling the creation of high-fidelity, temporally consistent avatars from easily acquired monocular videos, with efficient mobile deployment.

Methods

MoRF builds upon the Deferred Neural Rendering (DNR) framework, which uses a geometric proxy (SMPL-X body model) and a neural texture, rendered by an image-to-image network. The core novelty of MoRF lies in its neural texture warping technique. Instead of directly fitting the neural texture to potentially misaligned geometry across frames, MoRF learns a frame-specific 2D warping field in the texture space. This warping field effectively compensates for misalignments between the SMPL-X mesh and the actual clothing/body shape in each frame, making the learning task easier for the neural texture and renderer. This warping is only used during training, ensuring fast inference.

Key methodological contributions include:

  • Neural Texture Warping: A novel 2D warping field learned per frame to align the neural texture with the input image, mitigating issues from geometric proxy imperfections.
  • Refined SMPL-X Mesh Fitting: Improvements to the SMPL-X fitting process, including shared shape parameters, a silhouette loss, and a temporal loss on pose vectors, to achieve more accurate and consistent geometric proxies.
  • Spectral Texture Initialization: Using spectral initialization for the neural texture, which helps slow down overfitting and improves temporal consistency.
  • Mobile Rendering Optimization: Techniques for efficient rendering on mobile devices, including quantization of neural network weights.

Datasets

The paper uses the following datasets:

  • Self-captured dataset: A custom dataset of stationary monocular videos of ten individuals, designed to avoid complex poses and focus on slow movements and turns. Hold-out videos feature novel poses.
  • ZJU-MoCap dataset: A multi-view dataset used for evaluation, with both multi-view and monocular fitting approaches tested.
  • People Snapshot dataset: Used for qualitative comparisons.

Key Insights and Ideas

  • Texture space warping is more robust: Warping in the 2D texture space is found to be less ambiguous and easier to train than 3D warping, especially for monocular video inputs.
  • Geometric proxy imperfections are common: Even with advanced fitting, SMPL-X models struggle to perfectly capture the nuances of clothing and body shape, necessitating techniques like warping to compensate.
  • Temporal consistency is crucial for perceived quality: While metrics like SSIM and LPIPS are important, user studies reveal that temporal stability and absence of artifacts (like flickering or ghosting) significantly influence perceived avatar quality, even if pixel-wise metrics are slightly lower.
  • Spectral initialization aids regularization: Initializing the neural texture spectrally acts as a prior, slowing down overfitting and allowing the warping mechanism more time to converge, leading to better temporal consistency.

Key Results

  • Real-time Mobile Rendering: MoRF avatars can be rendered at 30 FPS on mobile devices (Qualcomm Snapdragon 888).
  • High Visual Quality: MoRF avatars achieve higher image sharpness and temporal consistency compared to prior monocular video-based avatar systems like StylePeople and ANR.
  • User Preference: In user studies, MoRF avatars were preferred over competing methods, with a preference rate of 96.5% against InstantAvatar, 90.0% against HF-Avatar, and 98.0% against AnimNeRF in one scenario.
  • Faster Training: Training takes a few hours on a single GPU.
  • Improved Mesh Fitting: The proposed mesh fitting enhancements significantly improve avatar quality, with a combined improvement leading to an 84.8% user preference rate over the baseline SMPLify-X.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Clothing Movement: MoRF does not explicitly model clothing movement or loose clothing dynamics.
  • Unseen Areas: There is no prior for rendering areas of the body not visible in the training video.
  • Texture Seams: Texture seams can still appear, and warping artifacts might occur there.

Future work suggested includes addressing texture seams by using spherical textures and exploring more advanced methods for handling clothing dynamics.

Summary updated

BibTeX
@inproceedings{bashirov2024morf,
  author = {Renat Bashirov and Alexey Larionov and Evgeniya Ustinova and Mikhail Sidorenko and David Svitov and Ilya Zakharkin and Victor Lempitsky},
  title = {{MoRF: Mobile Realistic Fullbody Avatars From a Monocular Video}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure