TL;DR
This paper introduces a novel end-to-end neural rendering architecture for creating controllable 3D human models from monocular videos. By combining graph convolutional networks (GCNs) for mesh deformation with differentiable rendering and adversarial training for texture generation, the method achieves high-quality, controllable human synthesis that can be reenacted in new contexts.
What Problem Does This Paper Address?
Synthesizing realistic and controllable 3D human models from 2D images or videos is a challenging task. Existing methods often struggle with accurately capturing fine geometric details, generating plausible textures, and offering fine-grained control over the synthesized human's pose and appearance. While 2D image-to-image translation methods can produce visually appealing results, they lack the explicit 3D representation needed for applications in virtual and augmented reality. Image-based rendering techniques can create 3D models but often require manual design and lack the learning capabilities to improve visual quality with more data. This paper aims to bridge this gap by developing a unified, end-to-end system that generates fully controllable 3D human models with both accurate shape and realistic texture.
Methods
The proposed architecture consists of three main components trained in an end-to-end fashion:
- Human Mesh Estimation (Mesh Refinement Network): This component uses a three-stage Graph Convolutional Network (GCN) to refine a generic 3D human mesh (initially obtained from the SMPL model) based on rendered silhouettes. The GCN leverages the human body's manifold structure and ensures vertex consistency, leading to a more detailed and accurate mesh. This stage is trained using a silhouette loss derived from differentiable rendering.
- Texture Network (Coarse Texture Generation): A U-Net architecture is employed to generate a coarse texture map. This network is conditioned on a visibility map, which highlights the parts of the texture that need to be generated for a realistic appearance. It's trained using an L1 loss on the rendered image.
- Texture Refinement Network (Detailed Texture Generation): This stage further refines the coarse texture to produce finer details. It uses an adversarial training approach, with a generator network (the Texture Refinement Network) and two discriminators (one for faces, one for the whole image). This adversarial setup encourages the generation of photo-realistic textures.
The entire pipeline is trained using a stage-wise strategy, starting with the mesh refinement, then the coarse texture generation, and finally the texture refinement with adversarial regularization. Differentiable rendering plays a crucial role throughout the process, enabling gradient propagation from 2D image outputs back to the 3D model parameters.
Datasets
The paper uses four-minute videos from a previous work [9] for training the texture models and the mesh deformation network. These videos feature actors performing random movements. For evaluation and comparison with state-of-the-art methods, they utilize publicly available videos previously used by Chan et al. [2], Liu et al. [24], and Gomes et al. [9].
Key Insights and Ideas
- Synergy of GCNs and Differentiable Rendering: The paper effectively combines the structural understanding of GCNs for mesh deformation with the gradient-based learning capabilities of differentiable rendering. This allows for precise control over shape refinement by directly optimizing for visual cues like silhouettes.
- Controllable 3D Representation: By generating a fully controllable 3D parametric model, the method moves beyond 2D image synthesis, enabling applications that require explicit control over pose, viewpoint, and appearance.
- Adversarial Texture Refinement: The use of adversarial training specifically for texture refinement is key to achieving photo-realistic details, overcoming limitations of purely L1-based texture generation.
- Stage-wise Training for Stability: The multi-stage training approach (mesh, then coarse texture, then refined texture) helps stabilize the learning process and allows each component to specialize before being integrated.
Key Results
- Quantitative Performance: The proposed method achieves state-of-the-art results across several metrics, including SSIM, LPIPS, MSE, and FVD, outperforming existing methods in most categories. For instance, in the "jump" motion type, their SSIM score is 0.890, which is higher than other methods.
- Qualitative Improvements: Visual comparisons demonstrate that the model generates more accurate body shapes and finer texture details, avoiding artifacts seen in other methods. The ablation studies show the importance of both vertex refinement clamping and adversarial texture refinement for achieving high-quality results.
- Efficiency: The inference time is reported to be 92 ms per frame, with 90 ms for mesh deformation and 2 ms for texture networks, making it practical for real-time applications.
Limitations and Future Work
- Limitations:
- The method relies on the SMPL model, which might not perfectly capture highly complex or non-humanoid body shapes.
- While the method handles different body shapes and clothing, extreme clothing deformations or highly intricate details might still pose challenges.
- The current implementation requires a silhouette for training the mesh refinement, which might not always be readily available in all scenarios.
- Future Work: The authors suggest that their controllable 3D representation opens doors for various applications in virtual and augmented reality, as well as for more advanced animation and reenactment tasks. Further improvements could focus on handling more complex clothing dynamics and reducing reliance on explicit silhouette supervision during training.
Summary updated