Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Guoxing Sun, Xin Chen, Yizhang Chen, Anqi Pang, Pei Lin, Yuheng Jiang, Lan Xu, Jingyi Yu, Jingya Wang

Presented at ACM MM 2021, this research introduces a novel system for high-quality 4D reconstruction and rendering of humans interacting with objects using a sparse setup of only six RGB cameras. By decoupling the scene into distinct layers and employing occlusion-aware neural networks, the system achieves photo-realistic results even under significant occlusions.

Introduction

Creating free-viewpoint video—allowing a virtual camera to move freely around a recorded scene—traditionally requires a dense array of synchronized cameras. The challenge intensifies during human-object interactions (HOI), where objects frequently obscure parts of the performer. Existing sparse-view methods often suffer from ghosting artifacts or blurry textures when faced with such complex occlusions. This paper addresses these limitations through a "layer-wise" scene decoupling approach.

Methodology

The core of the proposed pipeline is the separation of the human and the object into independent layers, allowing for specialized processing of each component.

The proposed neural rendering pipeline, featuring scene decoupling, occlusion-aware reconstruction, and direction-aware blending.

1. Occlusion-aware Implicit Human Reconstruction

The system utilizes a PIFu-based implicit function for geometry representation. To handle occlusions, it integrates pixel-aligned features with a global human motion prior derived from the SMPL parametric model. A key innovation is the use of synthetic occlusion training: during the learning phase, 3D objects from the ShapeNet dataset are randomly placed around human models to train the network to robustly reconstruct hidden areas.

2. Human-aware Object Tracking

Objects are treated as rigid bodies. After an initial template-based alignment, the system tracks the object's 6-degree-of-freedom (6DoF) pose using differentiable rendering. To ensure accuracy, the tracking algorithm explicitly ignores regions where the human mask overlaps the object, preventing interference from the performer's limbs.

3. Layered Rendering and Texture Blending

For final image synthesis, the system employs Direction-aware Neural Texture Blending. Instead of simple color averaging, it learns a blending weight map that accounts for camera angles and occlusion maps. For areas hidden in all current views, a Spatial-temporal Texture Completion module retrieves information from a canonical texture map built from previous frames.

Experimental Results

The system was evaluated on real-world sequences captured with a six-camera setup, involving various objects like boxes, balls, and chairs.

Experimental setup (a) and qualitative reconstruction results (b) showing various complex human-object interactions.

Quantitative Performance

The proposed method significantly outperforms state-of-the-art baselines like NeRF and NHR in sparse-view scenarios:

  • Image Fidelity: Achieved a Peak Signal-to-Noise Ratio (PSNR) of 25.323, compared to NeRF's 24.022.
  • Error Reduction: Reduced the Mean Absolute Error (MAE) to 4.787, nearly half that of NeRF (9.612) in interaction-heavy scenes.

Qualitative Quality

Visual results demonstrate much sharper clothing folds and well-defined object boundaries. Unlike previous methods, this system maintains clear separation between the performer's hands and the objects they manipulate, effectively avoiding the "bleeding" of geometry between layers.

Conclusion and Future Work

This research demonstrates that scene decoupling is a powerful strategy for 4D reconstruction from sparse views. While currently limited to rigid objects and constrained by the resolution of the camera setup, the approach sets a new benchmark for rendering complex human-object interactions. Future work will focus on handling non-rigid objects and improving the capture of fine details like facial expressions and finger movements.

Summary updated

BibTeX
@inproceedings{sun2021neural,
  author = {Guoxing Sun and Xin Chen and Yizhang Chen and Anqi Pang and Pei Lin and Yuheng Jiang and Lan Xu and Jingyi Yu and Jingya Wang},
  title = {{Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475442},
}
Figure