Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis

Tianhan Xu, Yasuhiro Fujita, Eiichi MatsumotoCVPR 2022 · DOI · arXiv

Abstract

Creating high-quality, controllable 3D human avatars from sparse multi-view videos remains a significant challenge. While Neural Radiance Fields (NeRF) offer photorealism, they often struggle with animatability and generalization to unseen poses. This paper introduces Surface-Aligned NeRF, a novel approach that builds radiance fields directly on a mesh surface. By replacing standard 3D coordinates with a surface-aligned representation and a specialized "dispersed projection" algorithm, the method achieves superior detail and better generalization without the need for complex, learned deformation fields.

Introduction

The synthesis of realistic, animatable 3D humans is a cornerstone of computer graphics. Existing methods typically rely on parametric models like SMPL for control or NeRF for photorealism. However, bridging these two—making NeRFs move—usually involves learning "deformation fields" to warp points between observation and canonical spaces. These fields are often difficult to train and fail on novel poses.

A key technical hurdle identified by the authors is the "indistinguishability issue": standard nearest-point projections from 3D space to a mesh surface are not injective. This means multiple distinct points in 3D space can map to the same point on the mesh surface, preventing the neural network from distinguishing between different spatial locations and leading to a loss of fine detail.

Methodology

Surface-Aligned NeRF bypasses learned deformation fields by using the geometry of a parametric human mesh (SMPL) as an explicit coordinate system.

Overview of the Surface-Aligned NeRF pipeline, showing the injective mapping from 3D space to surface-aligned coordinates and the subsequent MLP-based radiance field.

1. Surface-Aligned Representation

Instead of standard $(x, y, z)$ coordinates, points are represented as $(s_c, h)$, where $s_c$ is a point on a canonical T-pose mesh surface and $h$ is the signed distance (height) from that surface.

2. Dispersed Projection

To ensure the mapping is injective (one-to-one), the authors developed a "dispersed projection" algorithm. It uses barycentric interpolation to project points onto planes parallel to mesh triangles and performs vertex normal alignment. This ensures that even near sharp edges or complex geometry, the NeRF can distinguish between distinct spatial locations, preserving details like clothing folds.

3. Architecture

The model processes surface-aligned coordinates, view directions (in both world and local surface frames), and a latent skeleton pose code. This local coordinate awareness allows the model to better capture how light interacts with the body surface and how the surface deforms with pose changes.

Results

The method was evaluated on the ZJU-MoCap and Human3.6M datasets, consistently outperforming state-of-the-art approaches such as Neural Body and Animatable NeRF.

Qualitative comparison of the proposed method against state-of-the-art approaches. Our method preserves significantly more fine detail and structural consistency across different subjects.

Key results include:

  • State-of-the-Art Performance: The method improved average PSNR on the Human3.6M dataset from approximately 23.0 to 24.28.
  • Superior Generalization: The model maintains high fidelity when rendering "unseen" poses, demonstrating the robustness of anchoring the radiance field to explicit geometry.
  • Explicit Control: By leveraging the SMPL model, users can explicitly adjust body shape or swap clothing between subjects by segmenting the mesh.

Conclusion

Surface-Aligned NeRF demonstrates that anchoring radiance fields to explicit geometry provides a powerful prior for human synthesis. By ensuring an injective mapping to the mesh surface, the authors overcome fundamental limitations in detail preservation common in previous deformation-based methods. While the method currently depends on accurate initial mesh fitting, it represents a significant step toward realistic and controllable digital humans.

Summary updated

BibTeX
@inproceedings{xu2022surface,
  author = {Tianhan Xu and Yasuhiro Fujita and Eiichi Matsumoto},
  title = {{Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01542},
}
Figure