Abstract
Recent advances in neural human rendering have enabled high-fidelity novel view synthesis from monocular videos. However, existing state-of-the-art methods are typically identity-specific, requiring a separate model to be trained for every individual. This paper introduces YOTO (You Only Train Once), a unified framework designed to learn and render multiple human identities simultaneously. By leveraging learnable identity codes and a pose-conditioned query mechanism, YOTO achieves superior rendering quality while significantly reducing training time and storage requirements compared to per-identity models.
Introduction
Creating high-quality digital avatars from a single smartphone video is a cornerstone for applications in virtual and augmented reality. While methods like HumanNeRF have demonstrated impressive results, their lack of scalability is a major hurdle. Training a unique model for every user is computationally prohibitive and ignores the potential for shared anatomical knowledge across different subjects. Furthermore, isolated models cannot easily perform motion transfer—the ability to apply one person's movement to another's appearance.
YOTO addresses these challenges by proposing a collaborative learning framework. Instead of training $N$ models for $N$ people, YOTO trains a single model that captures a shared canonical representation of the human form, using identity-specific codes to modulate appearance and geometry.
Methodology
The YOTO framework is built upon three primary innovations that allow it to scale across multiple identities while maintaining high-frequency details.
Learnable Identity Codes
To distinguish between different subjects, YOTO introduces a set of $N$ learnable identity vectors $S \in \mathbb{R}^{N \times 256}$. These codes act as latent descriptors that inform the network about the specific geometry and texture of each individual. During training, the model learns to associate these codes with the unique features of each subject in the dataset.
Pose-Conditioned Code Query
Human appearance is highly dynamic; clothing deformations and skin folds are non-rigid and pose-dependent. YOTO employs a cross-attention mechanism between the identity code and the 3D joint positions of the subject. This allows the model to extract pose-specific features, effectively predicting how a particular person's clothing should wrinkle or shift in a given stance.
Unified NeRF Pipeline
The system maps points from the observation space (the video frame) to a canonical "T-pose" space using skeletal transformations. A non-rigid motion offset, conditioned by the pose-query, is applied to account for complex deformations. Finally, a canonical MLP predicts color and density, conditioned by the identity code to ensure subject-specific rendering.
Experiments and Datasets
The researchers evaluated YOTO on two standard benchmarks:
- ZJU-MoCap: Used for rigorous quantitative evaluation across 6 subjects.
- PeopleSnapshot: A dataset of 22 "in-the-wild" videos used to demonstrate the model's scalability to a larger number of identities in diverse environments.
Results and Discussion
YOTO demonstrates significant improvements over the previous state-of-the-art, HumanNeRF, in both efficiency and quality.
Rendering Quality
Quantitatively, YOTO achieves higher PSNR and lower LPIPS scores. Qualitatively, the model is more robust at reconstructing body parts that were poorly observed in the input video (e.g., the back of the torso) and handles complex clothing textures with greater precision.
Efficiency and Scalability
The collaborative nature of YOTO leads to massive gains in efficiency:
- Training Speed: Training 6 subjects simultaneously was 4.7x faster than training 6 individual HumanNeRF models (31 hours vs. 147 hours).
- Storage: The unified model for 6 identities requires only 747MB, an 83% reduction compared to the 4.4GB needed for separate models.
- Motion Transfer: Because all identities share a canonical space, motion transfer is achieved natively by applying the pose parameters of one subject to the identity code of another.
Limitations and Future Work
Despite its success, YOTO relies on accurate 3D pose estimation; errors in the initial pose can propagate to the final rendering. Future work may explore scaling the system to hundreds of identities and integrating dynamic backgrounds into the unified rendering pipeline.
Summary updated