Occupancy Planes for Single-view RGB-D Human Reconstruction

Xiaoming Zhao, Yuan-Ting Hu, Zhongzheng Ren, Alexander G. Schwing

TL;DR

This paper introduces "Occupancy Planes" (OPlanes), a novel representation for single-view RGB-D human reconstruction. By predicting occupancy on a series of planes slicing through the camera's view frustum, OPlanes effectively leverage spatial correlations, leading to more robust and accurate reconstructions, especially for partially visible or occluded humans.

What Problem Does This Paper Address?

Reconstructing a 3D human shape from a single RGB-D image is a challenging task due to the inherent ambiguity of inferring unseen parts of the body. Existing methods often rely on per-point classification within implicit function frameworks. While these methods can produce detailed results, they treat each 3D point's occupancy prediction independently. This isolation limits their ability to leverage correlations between neighboring points, leading to noisy predictions, particularly in difficult scenarios like partial occlusions or when only a portion of the human is visible. The paper aims to overcome this limitation by proposing a representation that inherently captures these spatial relationships.

Methods

The core contribution is the Occupancy Planes (OPlanes) representation. Instead of predicting occupancy for individual 3D points, OPlanes represent the human shape as a collection of 2D binary images (planes) that are fronto-parallel and slice through the camera's view frustum at various depth levels.

Here's a breakdown of the approach:

  • OPlanes Representation: A set of $N$ planes, $O^{H \times W}_{Z_N} = \{O^{H \times W}_{z_i} | z_i \in Z_N\}$, where each $O^{H \times W}_{z_i}$ is a binary image indicating occupancy at a specific depth $z_i$.
  • Ground Truth Generation: For training, ground truth occupancy values for each pixel on a plane are derived by unprojecting 2D image pixels to 3D coordinates and checking if they fall within the ground truth 3D mesh.
  • Network Architecture: A fully convolutional network, $f_{spatial}$, is used to predict these occupancy planes. It fuses image features (from a ResNet50-based FPN backbone and a custom RGB network) and depth features (derived from a depth difference image). The depth difference image encodes how far a point is from the human's visible surface.
  • Loss Functions: The model is trained end-to-end using a combination of:
  • High-resolution supervision ($L^{H \times W}_{HO \times WO}$): This includes Binary Cross-Entropy (BCE) and Dice loss applied to the final predicted occupancy planes.
  • Intermediate feature supervision ($L^{h \times w}_{ho \times wo}$): This uses BCE and Dice loss on intermediate feature maps, encouraging better correlation between image and depth features via an inner-product operation.
  • Inference: During inference, a dense voxel grid is constructed from the predicted OPlanes, and a marching cubes algorithm is used to extract the final 3D mesh. The number of planes used during inference can be higher than during training for improved detail.

Datasets

  • S3D (Hu et al., 2021): This synthetic dataset, built on GTA-V, was used for training and evaluation. It provides ground-truth meshes, masks, and depth maps. The authors highlight that S3D offers more diverse poses, occlusions, and appearances compared to previous datasets.
  • Real-world RGB-D data: Captured using a 2020 iPad Pro with a LiDAR sensor, this data was used for qualitative evaluation to demonstrate the model's ability to transfer to real-world scenarios without fine-tuning.

Key Insights and Ideas

  • Spatial Correlations are Key: The paper strongly argues that explicitly modeling spatial correlations between neighboring points on a plane is crucial for robust human reconstruction, outperforming independent per-point predictions.
  • Planes as a Flexible Representation: OPlanes offer a balance between the structured nature of voxel grids and the flexibility of implicit representations. They allow for adaptive resolution and better leverage of local context.
  • Depth Difference Feature: The use of a depth difference image as an input feature is insightful. It helps the network understand the geometry relative to the visible surface, which is particularly useful for inferring occluded regions.
  • Intermediate Supervision Matters: Supervising intermediate features, not just the final output, significantly improves performance, suggesting that guiding the feature learning process is beneficial.
  • Robustness to Partial Visibility: The OPlanes representation demonstrates a notable advantage in handling partially visible humans, a common failure case for prior methods.

Key Results

  • Quantitative Improvements: OPlanes achieved state-of-the-art results on the S3D dataset, outperforming PIFuHD and IF-Net across key metrics:
  • IoU (Intersection over Union): OPlanes achieved 0.691±0.013, significantly higher than PIFuHD (0.428) and IF-Net (0.584).
  • Chamfer Distance: OPlanes achieved 0.155±0.008, lower than PIFuHD (0.332) and IF-Net (0.216), indicating better completeness and accuracy.
  • Normal Consistency: OPlanes achieved 0.749±0.005, higher than PIFuHD (0.677) and IF-Net (0.738), suggesting better surface detail reconstruction.
  • Qualitative Success: Figures 3 and 4 showcase compelling qualitative results, demonstrating OPlanes' ability to reconstruct consistent geometry even for challenging poses and partial occlusions, where other methods struggle. The model also shows good transferability to real-world data.
  • Ablation Studies: Removing neighbor information (1x1 kernel) or intermediate supervision led to performance degradation, validating the importance of these design choices. Training with more planes also yielded marginal improvements.

Limitations and Future Work

  • Floating Artifacts: The authors acknowledge that floating artifacts behind visible surfaces can still occur, especially when inferring invisible parts. They use post-processing (smoothing functions) for visualization purposes but note this is not used during quantitative evaluation.
  • Visibility Dependence: While OPlanes are more robust to partial visibility than prior methods, performance still degrades as visibility decreases, as shown in Table S1.
  • Future Work: The paper suggests that the OPlanes representation is memory-efficient during training and can enable accurate reconstruction at inference time. Further exploration of adaptive plane selection and integration with other representations could be future directions.

Summary updated

Abstract

Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. Specifically, a set of 3D locations within the view-frustum of the camera are first projected independently onto the image and a corresponding feature is subsequently extracted for each 3D location. The feature of each 3D location is then used to classify independently whether the corresponding 3D point is inside or outside the observed object. This procedure leads to sub-optimal results because correlations between predictions for neighboring locations are only taken into account implicitly via the extracted features. For more accurate results we propose the occupancy planes (OPlanes) representation, which enables to formulate single-view RGB-D human reconstruction as occupancy prediction on planes which slice through the camera's view frustum. Such a representation provides more flexibility than voxel grids and enables to better leverage correlations than per-point classification. On the challenging S3D data we observe a simple classifier based on the OPlanes representation to yield compelling results, especially in difficult situations with partial occlusions due to other objects and partial visibility, which haven't been addressed by prior work.
BibTeX
@inproceedings{zhao2023occupancy,
  author = {Xiaoming Zhao and Yuan-Ting Hu and Zhongzheng Ren and Alexander G. Schwing},
  title = {{Occupancy Planes for Single-view RGB-D Human Reconstruction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=gFvypcqKG5},
}
Figure