Presented at ACM MM 2021, this research introduces a novel system for high-quality 4D reconstruction and rendering of humans interacting with objects using a sparse setup of only six RGB cameras. By decoupling the scene into distinct layers and employing occlusion-aware neural networks, the system achieves photo-realistic results even under significant occlusions.
Introduction
Creating free-viewpoint video—allowing a virtual camera to move freely around a recorded scene—traditionally requires a dense array of synchronized cameras. The challenge intensifies during human-object interactions (HOI), where objects frequently obscure parts of the performer. Existing sparse-view methods often suffer from ghosting artifacts or blurry textures when faced with such complex occlusions. This paper addresses these limitations through a "layer-wise" scene decoupling approach.
Methodology
The core of the proposed pipeline is the separation of the human and the object into independent layers, allowing for specialized processing of each component.
1. Occlusion-aware Implicit Human Reconstruction
The system utilizes a PIFu-based implicit function for geometry representation. To handle occlusions, it integrates pixel-aligned features with a global human motion prior derived from the SMPL parametric model. A key innovation is the use of synthetic occlusion training: during the learning phase, 3D objects from the ShapeNet dataset are randomly placed around human models to train the network to robustly reconstruct hidden areas.
2. Human-aware Object Tracking
Objects are treated as rigid bodies. After an initial template-based alignment, the system tracks the object's 6-degree-of-freedom (6DoF) pose using differentiable rendering. To ensure accuracy, the tracking algorithm explicitly ignores regions where the human mask overlaps the object, preventing interference from the performer's limbs.
3. Layered Rendering and Texture Blending
For final image synthesis, the system employs Direction-aware Neural Texture Blending. Instead of simple color averaging, it learns a blending weight map that accounts for camera angles and occlusion maps. For areas hidden in all current views, a Spatial-temporal Texture Completion module retrieves information from a canonical texture map built from previous frames.
Experimental Results
The system was evaluated on real-world sequences captured with a six-camera setup, involving various objects like boxes, balls, and chairs.
Quantitative Performance
The proposed method significantly outperforms state-of-the-art baselines like NeRF and NHR in sparse-view scenarios:
- Image Fidelity: Achieved a Peak Signal-to-Noise Ratio (PSNR) of 25.323, compared to NeRF's 24.022.
- Error Reduction: Reduced the Mean Absolute Error (MAE) to 4.787, nearly half that of NeRF (9.612) in interaction-heavy scenes.
Qualitative Quality
Visual results demonstrate much sharper clothing folds and well-defined object boundaries. Unlike previous methods, this system maintains clear separation between the performer's hands and the objects they manipulate, effectively avoiding the "bleeding" of geometry between layers.
Conclusion and Future Work
This research demonstrates that scene decoupling is a powerful strategy for 4D reconstruction from sparse views. While currently limited to rigid objects and constrained by the resolution of the camera setup, the approach sets a new benchmark for rendering complex human-object interactions. Future work will focus on handling non-rigid objects and improving the capture of fine details like facial expressions and finger movements.
Summary updated