You Only Train Once: Multi-Identity Free-Viewpoint Neural Human Rendering from Monocular Videos

Jaehyeok Kim, Dongyoon Wee, Dan XuarXiv 2023 · DOI · arXiv

Abstract

Recent advances in neural human rendering have enabled high-fidelity novel view synthesis from monocular videos. However, existing state-of-the-art methods are typically identity-specific, requiring a separate model to be trained for every individual. This paper introduces YOTO (You Only Train Once), a unified framework designed to learn and render multiple human identities simultaneously. By leveraging learnable identity codes and a pose-conditioned query mechanism, YOTO achieves superior rendering quality while significantly reducing training time and storage requirements compared to per-identity models.

Introduction

Creating high-quality digital avatars from a single smartphone video is a cornerstone for applications in virtual and augmented reality. While methods like HumanNeRF have demonstrated impressive results, their lack of scalability is a major hurdle. Training a unique model for every user is computationally prohibitive and ignores the potential for shared anatomical knowledge across different subjects. Furthermore, isolated models cannot easily perform motion transfer—the ability to apply one person's movement to another's appearance.

YOTO addresses these challenges by proposing a collaborative learning framework. Instead of training $N$ models for $N$ people, YOTO trains a single model that captures a shared canonical representation of the human form, using identity-specific codes to modulate appearance and geometry.

Methodology

The YOTO framework is built upon three primary innovations that allow it to scale across multiple identities while maintaining high-frequency details.

Learnable Identity Codes

To distinguish between different subjects, YOTO introduces a set of $N$ learnable identity vectors $S \in \mathbb{R}^{N \times 256}$. These codes act as latent descriptors that inform the network about the specific geometry and texture of each individual. During training, the model learns to associate these codes with the unique features of each subject in the dataset.

Pose-Conditioned Code Query

Human appearance is highly dynamic; clothing deformations and skin folds are non-rigid and pose-dependent. YOTO employs a cross-attention mechanism between the identity code and the 3D joint positions of the subject. This allows the model to extract pose-specific features, effectively predicting how a particular person's clothing should wrinkle or shift in a given stance.

Overview of the YOTO architecture, illustrating the mapping from observation spaces to a shared canonical space via pose-conditioned identity codes.

Unified NeRF Pipeline

The system maps points from the observation space (the video frame) to a canonical "T-pose" space using skeletal transformations. A non-rigid motion offset, conditioned by the pose-query, is applied to account for complex deformations. Finally, a canonical MLP predicts color and density, conditioned by the identity code to ensure subject-specific rendering.

Experiments and Datasets

The researchers evaluated YOTO on two standard benchmarks:

  • ZJU-MoCap: Used for rigorous quantitative evaluation across 6 subjects.
  • PeopleSnapshot: A dataset of 22 "in-the-wild" videos used to demonstrate the model's scalability to a larger number of identities in diverse environments.

Results and Discussion

YOTO demonstrates significant improvements over the previous state-of-the-art, HumanNeRF, in both efficiency and quality.

Rendering Quality

Quantitatively, YOTO achieves higher PSNR and lower LPIPS scores. Qualitatively, the model is more robust at reconstructing body parts that were poorly observed in the input video (e.g., the back of the torso) and handles complex clothing textures with greater precision.

Qualitative comparison between HumanNeRF and YOTO. YOTO demonstrates superior reconstruction of fine details and unseen body parts, as highlighted by the red and green markers.

Efficiency and Scalability

The collaborative nature of YOTO leads to massive gains in efficiency:

  • Training Speed: Training 6 subjects simultaneously was 4.7x faster than training 6 individual HumanNeRF models (31 hours vs. 147 hours).
  • Storage: The unified model for 6 identities requires only 747MB, an 83% reduction compared to the 4.4GB needed for separate models.
  • Motion Transfer: Because all identities share a canonical space, motion transfer is achieved natively by applying the pose parameters of one subject to the identity code of another.

Limitations and Future Work

Despite its success, YOTO relies on accurate 3D pose estimation; errors in the initial pose can propagate to the final rendering. Future work may explore scaling the system to hundreds of identities and integrating dynamic backgrounds into the unified rendering pipeline.

Summary updated

BibTeX
@misc{kim2023you,
  author = {Jaehyeok Kim and Dongyoon Wee and Dan Xu},
  title = {{You Only Train Once: Multi-Identity Free-Viewpoint Neural Human Rendering from Monocular Videos}},
  year = {2023},
  eprint = {2303.05835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.05835},
}
Figure