Abstract
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.
Introduction
Creating realistic 3D digital humans is a cornerstone of gaming, movies, and telepresence. However, traditional methods often face a trade-off between speed and realism. Standard "linear blend skinning" is computationally efficient but results in rigid characters that lack secondary dynamics like cloth wrinkles or the sway of loose garments. Conversely, physics-based simulations offer high realism but are computationally expensive and difficult to tune for real-time applications.
This paper introduces a deep learning framework that generates video-realistic 3D human characters capable of real-time performance (38 FPS). By leveraging multi-view video for weak supervision, the model learns to predict complex, motion-dependent clothing dynamics and view-dependent textures without requiring expensive 3D scans or manual physics simulations.
Methodology
The proposed system utilizes a coarse-to-fine differentiable character representation trained in an "analysis-by-synthesis" loop. The pipeline consists of three primary neural modules:
- EGNet (Coarse Deformation): This module employs a novel Structure Aware Graph Convolution (SAGC) to predict the rotation and translation of an "embedded graph" (a simplified mesh). This captures large-scale movements, such as the swing of a dress.
- DeltaNet (Fine Deformation): This network regresses per-vertex displacements on top of the coarse mesh to capture fine-scale details like wrinkles.
- TexNet (Dynamic Texture): A UNet-based architecture produces a dynamic texture map, accounting for motion-dependent wrinkles and view-dependent effects like specular highlights.
Structure Aware Graph Convolution (SAGC)
A key innovation is the SAGC, which addresses the limitation of standard graph convolutions that apply the same filter to every node. Since different regions of a character (e.g., a knee vs. a flowing skirt) behave differently, SAGC assigns individual, learnable kernels to each node, allowing the network to learn region-specific deformation behaviors.
Weak Supervision and Canonical Learning
The model is trained using differentiable rendering, comparing rendered characters against original video frames via silhouette and chroma losses. By regressing deformations in a "canonical" T-pose space before applying skeletal posing, the network generalizes effectively to unseen motions.
Experiments and Results
The authors evaluated their method using a new benchmark dataset, DynaCap, featuring 5 sequences of 4 subjects in various apparel recorded at 25fps with up to 101 synchronized cameras. They also tested on the DeepCap dataset to demonstrate performance with sparser camera setups (11-14 views).
Key Performance Metrics
- Real-time Performance: The system achieves 38 FPS on a single Nvidia RTX 8000.
- Accuracy: The method reached 90.70% in foreground mask overlap (AMVIoU), significantly outperforming standard skinning (79.45%) and previous optimization-based methods (88.14%).
- Generalization: The model successfully synthesizes realistic dynamics for unseen motions, such as jumping jacks or complex dance moves, maintaining physical plausibility.
Discussion and Limitations
While the model achieves high-quality results in real-time, some limitations remain. The current representation does not track detailed facial expressions or hand movements and relies on a fixed template mesh, preventing it from handling topological changes like a character removing a jacket. Additionally, while inference is real-time, training requires several days on high-end hardware and is sensitive to the quality of foreground segmentation masks. Future work may address these areas to further enhance the versatility of deep dynamic characters.
Summary updated