Abstract
Single-view 3D human reconstruction is a challenging ill-posed problem due to depth ambiguities and the lack of global structural constraints. We present Geo-PIFu, a method that advances state-of-the-art reconstruction by integrating 2D pixel-aligned features with a 3D geometry-aligned latent voxel representation. This hybrid approach effectively resolves depth ambiguities along camera rays and enforces global structural consistency. Our results on the DeepHuman dataset demonstrate significant improvements in both global accuracy and local detail, producing high-fidelity meshes even in complex poses and occluded regions.
1. Introduction
Reconstructing a 3D clothed human from a single 2D image is inherently ill-posed, as multiple 3D shapes can project to the same 2D image. Previous state-of-the-art methods like Pixel-aligned Implicit Functions (PIFu) have achieved impressive results by learning local implicit functions. However, PIFu suffers from two primary limitations: feature ambiguity, where points at different depths project to the same pixel, and a lack of global robustness, leading to unrealistic poses or broken topology in self-occluded areas. Geo-PIFu addresses these issues by introducing a 3D "geometry-aligned" branch that provides a coarse structural proxy to guide the high-resolution reconstruction.
2. Methodology
Geo-PIFu utilizes a dual-branch architecture to capture both high-frequency local details and robust global structure.
2.1 Dual-Branch Feature Extraction
- 2D Pixel-Aligned Branch: Similar to PIFu, a 2D U-Net extracts high-frequency features (e.g., clothing wrinkles) from the input image.
- 3D Geometry-Aligned Branch: A structure-aware 3D U-Net "lifts" image features into a 3D latent voxel space. These voxels serve as a coarse human shape proxy, providing essential global context.
2.2 Feature Fusion and Occupancy Prediction
For any query point $P$ in 3D space, the system performs trilinear interpolation on the 3D voxel grid and bilinear interpolation on the 2D image features. These features are concatenated with the point's depth and processed by a Multi-Layer Perceptron (MLP) to predict occupancy $\sigma(P)$.
2.3 Supervision
The model is trained using a multi-task loss. One loss supervises a coarse occupancy volume (ensuring the 3D voxels capture the general human shape), while the other supervises the high-resolution implicit surface function. Interestingly, the 3D decoder used to supervise the voxels is only needed during training; at inference, the model directly interpolates from the latent voxel features.
3. Experimental Setup
The model is evaluated on the DeepHuman dataset, which consists of 5,436 training and 1,359 test meshes. This dataset is 10x larger than the private commercial dataset used in the original PIFu paper and is significantly more diverse, featuring complex poses, varied clothing, and free camera rotations.
4. Results and Discussion
Geo-PIFu demonstrates substantial quantitative and qualitative improvements over existing methods.
4.1 Quantitative Performance
- Global Accuracy: Geo-PIFu achieved a 42.7% reduction in Chamfer Distance and Point-to-Surface Distance (PSD) compared to PIFu.
- Local Detail: Normal estimation errors were reduced by 19.4%, indicating superior capture of fine fabric details.
- Efficiency: The model is 12x smaller in parameter count than PIFuHD while maintaining higher performance on the DeepHuman benchmark.
4.2 Qualitative Analysis
As shown in Figure 2, Geo-PIFu produces much cleaner reconstructions, particularly for the "invisible" back-side of subjects and complex extremities like hands and feet. The latent voxels act as a regularizer, preventing the model from "hallucinating" impossible body parts in occluded regions and resolving depth ambiguities along camera rays.
5. Conclusion
Geo-PIFu successfully fuses 2D and 3D representations to overcome the limitations of purely pixel-aligned methods. By incorporating a geometry-aligned branch, the model achieves unprecedented structural consistency and detail in single-view human reconstruction. Future work may explore combining this 3D awareness with high-resolution patch-based processing to further enhance reconstruction fidelity.
Summary updated