Abstract
Unsupervised learning of 3D-aware generative adversarial networks has lately made much progress. Some recent work demonstrates promising results of learning human generative models using neural articulated radiance fields, yet their generalization ability and controllability lag behind parametric human models, i.e., they do not perform well when generalizing to novel pose/shape and are not part controllable. To solve these problems, we propose VeRi3D, a generative human vertex-based radiance field parameterized by vertices of the parametric human template, SMPL. We map each 3D point to the local coordinate system defined on its neighboring vertices, and use the corresponding vertex feature and local coordinates for mapping it to color and density values. We demonstrate that our simple approach allows for generating photorealistic human images with free control over camera pose, human pose, shape, as well as enabling part-level editing.
Introduction
Generating realistic, 3D-controllable images of clothed humans is a major goal for virtual try-on, VR/AR, and digital content creation. While 3D-aware Generative Adversarial Networks (GANs) have succeeded with rigid objects, humans are articulated—they bend, stretch, and change pose. Existing methods often struggle with "ghosting" artifacts because mapping 3D points from an observation space back to a global canonical space is difficult to learn from 2D images alone. Furthermore, most models lack part-level controllability, preventing independent edits to specific items like pants or hairstyles.
VeRi3D addresses these issues by anchoring 3D image synthesis to the vertices of the SMPL parametric human model. By mapping 3D points to local coordinate systems defined by these vertices, the method achieves high photorealism and fine-grained control over body shape, pose, and part-level appearance.
Methodology
The core innovation of VeRi3D is the Vertex-based Radiance Field, which moves away from a global canonical space and instead uses a localized representation.
The pipeline consists of several key components:
- Vertex Parameterization: The system uses the SMPL model as a template, attaching feature vectors to each mesh vertex.
- Local Coordinate Systems: For any 3D point queried during volume rendering, the system identifies its $K$ nearest SMPL vertices. The point is transformed into the local coordinate systems of these vertices, making the representation inherently pose-agnostic.
- Feature Generation: A StyleGAN2-based 2D CNN generates a feature map, which is mapped onto the 3D vertices using a fixed UV mapping.
- The Decoder: An MLP takes aggregated vertex features and local coordinates to predict color and density. Because the features are tied to specific vertices, the model learns a pose-agnostic representation of human appearance.
- Part-level Editing: The SMPL mesh is segmented into parts (head, upper body, lower body). Applying Principal Component Analysis (PCA) to the vertex features of each part allows users to independently manipulate appearance and geometry.
Experimental Results
VeRi3D was evaluated on several prominent benchmarks, including DeepFashion, AIST++, and Surreal.
Quantitative Performance
VeRi3D achieved a Frechet Inception Distance (FID) of 21.4 on DeepFashion, significantly outperforming previous state-of-the-art models like ENARF (FID 60.3) and EG3D (FID 27.2). In terms of pose accuracy, the model demonstrated high fidelity to input poses, even for complex dance moves in the AIST++ dataset.
Qualitative Comparison
The model produces significantly sharper images with fewer artifacts compared to baselines, maintaining anatomical consistency across a wide range of poses.
 Source, p. 6
Controllability
The latent space of vertex features is semantically organized. PCA-based editing enables smooth transitions in body shape and successful part-level modifications, such as changing a hairstyle without altering the rest of the outfit.
Conclusion and Future Work
VeRi3D demonstrates that anchoring radiance fields to local vertex coordinates avoids the mapping problems that cause ghosting in previous articulated models. While the model currently depends on the quality of input pose estimations, future work may involve jointly refining pose distributions during training to mitigate noise in real-world data.
Summary updated