Point-Based Radiance Fields for Controllable Human Motion Synthesis

Haitao Yu, Deheng Zhang, Peiyuan Xie, Tianyi ZhangarXiv 2023 · OpenReview

Abstract

This paper proposes a novel controllable human motion synthesis method for fine-level deformation based on static point-based radiance fields. Although previous editable neural radiance field methods can generate impressive results on novel-view synthesis and allow naive deformation, few algorithms can achieve complex 3D human editing such as forward kinematics. Our method exploits the explicit point cloud to train the static 3D scene and apply the deformation by encoding the point cloud translation using a deformation MLP. To make sure the rendering result is consistent with the canonical space training, we estimate the local rotation using SVD and interpolate the per-point rotation to the query view direction of the pre-trained radiance field. Extensive experiments show that our approach can significantly outperform the state-of-the-art on fine-level complex deformation which can be generalized to other 3D characters besides humans.

Introduction

Standard Neural Radiance Fields (NeRFs) excel at capturing static scenes but struggle with dynamic motion. Existing editable NeRF methods often rely on rigid skeletal priors like the SMPL model, which limits their application to human subjects and fails for arbitrary creatures. Furthermore, deforming a 3D object often leads to view-inconsistency; if points are moved without accounting for their new orientation, specular highlights and textures fail to align with the new pose. This work introduces a model-agnostic, view-consistent system for animating 3D characters using point-based radiance fields.

Methods

The proposed pipeline consists of three main stages: static training, deformation, and ray-bending.

  1. Static Training: A PointNeRF is trained on a static character using multi-view images, creating a dense point cloud where each point carries a neural feature describing its appearance.
  2. Deformation MLP: A Multi-Layer Perceptron (MLP), inspired by Dynamic Point Fields (DPF), calculates point translations based on user-provided keypoints. This allows for complex motions without a pre-defined skeleton.
  3. Ray-Bending: This core innovation ensures lighting consistency. The system "bends" camera rays back to the canonical pose by estimating local rotation via Singular Value Decomposition (SVD) of point neighborhoods. The resulting rotation matrix transforms the query view direction, ensuring view-dependent effects like specular highlights remain accurate.
Overview of the Point-Based Radiance Fields pipeline, illustrating dataset generation, canonical space training, and deformed space point editing with ray bending.

Results and Evaluation

The authors evaluated the method on several synthetic datasets created with BlenderNeRF, including humanoids (e.g., Man, Woman, Spiderman) and general characters (e.g., Robot, Turtle, Dragon, Parrot).

Key findings include:

  • Superior Detail: The ray-bending technique preserves specular reflections and fine textures significantly better than methods that ignore rotation. As seen in qualitative comparisons, the proposed method avoids the "blurring" common in other deformation techniques.
  • Generalization: The pipeline successfully animates non-humanoid characters without requiring specific skeletal models, proving its versatility across different character types.
  • Quantitative Performance: While PSNR is comparable for diffuse objects, the visual quality for complex materials is markedly higher when ray bending is employed.
Qualitative comparison showing the impact of ray bending on rendering quality compared to ground truth and a baseline without ray bending.

Conclusion

This research presents a significant advancement in animatable 3D avatars by decoupling motion from appearance through point-based radiance fields. While the method demonstrates high-fidelity motion synthesis, it remains sensitive to the quality of the initial point cloud and faces challenges with real-world noisy captures. Future work may involve more end-to-end approaches to mitigate the precision requirements of off-the-shelf MVS models and improve background integration.

Summary updated

Abstract

This paper proposes a novel controllable human motion synthesis method for fine-level deformation based on static point-based radiance fields. Although previous editable neural radiance field methods can generate impressive results on novel-view synthesis and allow naive deformation, few algorithms can achieve complex 3D human editing such as forward kinematics. Our method exploits the explicit point cloud to train the static 3D scene and apply the deformation by encoding the point cloud translation using a deformation MLP. To make sure the rendering result is consistent with the canonical space training, we estimate the local rotation using SVD and interpolate the per-point rotation to the query view direction of the pre-trained radiance field. Extensive experiments show that our approach can significantly outperform the state-of-the-art on fine-level complex deformation which can be generalized to other 3D characters besides humans.
BibTeX
@misc{yu2023point,
  author = {Haitao Yu and Deheng Zhang and Peiyuan Xie and Tianyi Zhang},
  title = {{Point-Based Radiance Fields for Controllable Human Motion Synthesis}},
  year = {2023},
  eprint = {2310.03375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.03375},
}
Figure