Abstract
Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. Specifically, a set of 3D locations within the view-frustum of the camera are first projected independently onto the image and a corresponding feature is subsequently extracted for each 3D location. The feature of each 3D location is then used to classify independently whether the corresponding 3D point is inside or outside the observed object. This procedure leads to sub-optimal results because correlations between predictions for neighboring locations are only taken into account implicitly via the extracted features. For more accurate results we propose the occupancy planes (OPlanes) representation, which enables to formulate single-view RGB-D human reconstruction as occupancy prediction on planes which slice through the camera's view frustum. Such a representation provides more flexibility than voxel grids and enables to better leverage correlations than per-point classification. On the challenging S3D data we observe a simple classifier based on the OPlanes representation to yield compelling results, especially in difficult situations with partial occlusions due to other objects and partial visibility, which haven't been addressed by prior work.
Introduction
Reconstructing a 3D human model from a single image is a classic "ill-posed" problem in computer vision. While implicit function methods like PIFu have made significant progress, they often struggle when the subject is partially occluded or in a non-standard pose. Most state-of-the-art methods rely on per-point classification, querying thousands of individual 3D coordinates independently. This approach fails to explicitly account for spatial correlations—for instance, the high probability that a point adjacent to the torso is also part of the torso. Consequently, reconstructions often suffer from noise, "holes," or total failure in challenging scenarios like partial visibility behind furniture.
Methodology: The Occupancy Planes Representation
To address these limitations, we introduce Occupancy Planes (OPlanes). This representation treats 3D human reconstruction as a series of 2D occupancy predictions on planes slicing through the camera's view frustum. This approach offers a pragmatic middle ground between the memory intensity of voxel grids and the lack of structure in per-point implicit functions.
Reconstruction Pipeline
- Feature Extraction: An RGB-D image is processed through a Feature Pyramid Network (FPN) to extract multi-scale image features.
- Depth Encoding: For a query plane at depth $z_i$, the model calculates a "depth difference" map relative to the observed surface.
- Spatial Prediction ($f_{spatial}$): A fully convolutional network (CNN) processes the concatenated image and depth features to predict an entire 2D occupancy map for the plane. The CNN naturally enforces spatial consistency by considering neighboring pixels.
- Mesh Generation: Occupancy is predicted for 256 planes, which are stacked to form a volume. The final 3D mesh is extracted using the Marching Cubes algorithm.
The model is trained using a combination of Binary Cross Entropy (BCE) and DICE loss to manage the inherent class imbalance between empty space and the human volume.
Experiments and Results
We evaluated OPlanes on the S3D (Synthetic 3D) dataset, which features humans in realistic, non-centered poses (e.g., sitting, running, or partially occluded). We also tested generalization on real-world data captured with a LiDAR-equipped iPad Pro.
OPlanes significantly outperformed baselines such as PIFuHD and IF-Net. On the S3D dataset, OPlanes achieved an Intersection over Union (IoU) of 0.691, compared to PIFuHD’s 0.428. Furthermore, the Chamfer-$L_1$ Distance was reduced to 0.155, nearly half the error of PIFuHD (0.332). Qualitative analysis shows that while PIFuHD often produces "shattered" or fragmented meshes in complex poses, OPlanes maintains smooth, consistent geometry that faithfully matches the visible portions of the subject.
Discussion and Conclusion
The OPlanes representation demonstrates that leveraging 2D spatial context through plane-based prediction is superior to independent point-wise classification. It provides flexibility, as the model can be sampled at any depth resolution during inference.
However, limitations remain. Single-view reconstruction still requires the network to "hallucinate" the back of the subject, and the current implementation relies on a heuristic depth range. Additionally, the requirement for RGB-D input limits its use with standard 2D legacy photos. Future work could explore integrating depth estimation to broaden applicability. Overall, OPlanes offers a robust framework for high-fidelity 3D human reconstruction in unconstrained environments.
Summary updated