VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis

Xinya Chen, Jiaxin Huang, Yanrui Bin, Lu Yu, Yiyi LiaoICCV 2023 · DOI · OpenReview

Abstract

Unsupervised learning of 3D-aware generative adversarial networks has lately made much progress. Some recent work demonstrates promising results of learning human generative models using neural articulated radiance fields, yet their generalization ability and controllability lag behind parametric human models, i.e., they do not perform well when generalizing to novel pose/shape and are not part controllable. To solve these problems, we propose VeRi3D, a generative human vertex-based radiance field parameterized by vertices of the parametric human template, SMPL. We map each 3D point to the local coordinate system defined on its neighboring vertices, and use the corresponding vertex feature and local coordinates for mapping it to color and density values. We demonstrate that our simple approach allows for generating photorealistic human images with free control over camera pose, human pose, shape, as well as enabling part-level editing.

Introduction

Generating realistic, 3D-controllable images of clothed humans is a major goal for virtual try-on, VR/AR, and digital content creation. While 3D-aware Generative Adversarial Networks (GANs) have succeeded with rigid objects, humans are articulated—they bend, stretch, and change pose. Existing methods often struggle with "ghosting" artifacts because mapping 3D points from an observation space back to a global canonical space is difficult to learn from 2D images alone. Furthermore, most models lack part-level controllability, preventing independent edits to specific items like pants or hairstyles.

VeRi3D addresses these issues by anchoring 3D image synthesis to the vertices of the SMPL parametric human model. By mapping 3D points to local coordinate systems defined by these vertices, the method achieves high photorealism and fine-grained control over body shape, pose, and part-level appearance.

Methodology

The core innovation of VeRi3D is the Vertex-based Radiance Field, which moves away from a global canonical space and instead uses a localized representation.

Overview of the VeRi3D architecture, showing the mapping of 2D features to 3D SMPL vertices and the local coordinate query system.

The pipeline consists of several key components:

  1. Vertex Parameterization: The system uses the SMPL model as a template, attaching feature vectors to each mesh vertex.
  2. Local Coordinate Systems: For any 3D point queried during volume rendering, the system identifies its $K$ nearest SMPL vertices. The point is transformed into the local coordinate systems of these vertices, making the representation inherently pose-agnostic.
  3. Feature Generation: A StyleGAN2-based 2D CNN generates a feature map, which is mapped onto the 3D vertices using a fixed UV mapping.
  4. The Decoder: An MLP takes aggregated vertex features and local coordinates to predict color and density. Because the features are tied to specific vertices, the model learns a pose-agnostic representation of human appearance.
  5. Part-level Editing: The SMPL mesh is segmented into parts (head, upper body, lower body). Applying Principal Component Analysis (PCA) to the vertex features of each part allows users to independently manipulate appearance and geometry.

Experimental Results

VeRi3D was evaluated on several prominent benchmarks, including DeepFashion, AIST++, and Surreal.

Quantitative Performance

VeRi3D achieved a Frechet Inception Distance (FID) of 21.4 on DeepFashion, significantly outperforming previous state-of-the-art models like ENARF (FID 60.3) and EG3D (FID 27.2). In terms of pose accuracy, the model demonstrated high fidelity to input poses, even for complex dance moves in the AIST++ dataset.

Qualitative Comparison

The model produces significantly sharper images with fewer artifacts compared to baselines, maintaining anatomical consistency across a wide range of poses.

![Qualitative comparison of VeRi3D against ENARF and Tri.

  • Surf. baselines across various outfits and poses.](/figures/art_2e83aa33472a45b98e8bca4673aa3638) Source, p. 6

Controllability

The latent space of vertex features is semantically organized. PCA-based editing enables smooth transitions in body shape and successful part-level modifications, such as changing a hairstyle without altering the rest of the outfit.

Conclusion and Future Work

VeRi3D demonstrates that anchoring radiance fields to local vertex coordinates avoids the mapping problems that cause ghosting in previous articulated models. While the model currently depends on the quality of input pose estimations, future work may involve jointly refining pose distributions during training to mitigate noise in real-world data.

Summary updated

Abstract

Unsupervised learning of 3D-aware generative adversarial networks has lately made much progress. Some recent work demonstrates promising results of learning human generative models using neural articulated radiance fields, yet their generalization ability and controllability lag behind parametric human models, i.e., they do not perform well when generalizing to novel pose/shape and are not part controllable. To solve these problems, we propose VeRi3D, a generative human vertex-based radiance field parameterized by vertices of the parametric human template, SMPL. We map each 3D point to the local coordinate system defined on its neighboring vertices, and use the corresponding vertex feature and local coordinates for mapping it to color and density values. We demonstrate that our simple approach allows for generating photorealistic human images with free control over camera pose, human pose, shape, as well as enabling part-level editing.
BibTeX
@misc{chen2023veri,
  author = {Xinya Chen and Jiaxin Huang and Yanrui Bin and Lu Yu and Yiyi Liao},
  title = {{VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis}},
  year = {2023},
  eprint = {2309.04800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04800},
}
Figure