Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction

Tong He, John P. Collomosse, Hailin Jin, Stefano Soatto

Abstract

Single-view 3D human reconstruction is a challenging ill-posed problem due to depth ambiguities and the lack of global structural constraints. We present Geo-PIFu, a method that advances state-of-the-art reconstruction by integrating 2D pixel-aligned features with a 3D geometry-aligned latent voxel representation. This hybrid approach effectively resolves depth ambiguities along camera rays and enforces global structural consistency. Our results on the DeepHuman dataset demonstrate significant improvements in both global accuracy and local detail, producing high-fidelity meshes even in complex poses and occluded regions.

1. Introduction

Reconstructing a 3D clothed human from a single 2D image is inherently ill-posed, as multiple 3D shapes can project to the same 2D image. Previous state-of-the-art methods like Pixel-aligned Implicit Functions (PIFu) have achieved impressive results by learning local implicit functions. However, PIFu suffers from two primary limitations: feature ambiguity, where points at different depths project to the same pixel, and a lack of global robustness, leading to unrealistic poses or broken topology in self-occluded areas. Geo-PIFu addresses these issues by introducing a 3D "geometry-aligned" branch that provides a coarse structural proxy to guide the high-resolution reconstruction.

2. Methodology

Geo-PIFu utilizes a dual-branch architecture to capture both high-frequency local details and robust global structure.

The Geo-PIFu architecture combines a 2D pixel-aligned branch with a 3D geometry-aligned branch to predict query point occupancy.

2.1 Dual-Branch Feature Extraction

  • 2D Pixel-Aligned Branch: Similar to PIFu, a 2D U-Net extracts high-frequency features (e.g., clothing wrinkles) from the input image.
  • 3D Geometry-Aligned Branch: A structure-aware 3D U-Net "lifts" image features into a 3D latent voxel space. These voxels serve as a coarse human shape proxy, providing essential global context.

2.2 Feature Fusion and Occupancy Prediction

For any query point $P$ in 3D space, the system performs trilinear interpolation on the 3D voxel grid and bilinear interpolation on the 2D image features. These features are concatenated with the point's depth and processed by a Multi-Layer Perceptron (MLP) to predict occupancy $\sigma(P)$.

2.3 Supervision

The model is trained using a multi-task loss. One loss supervises a coarse occupancy volume (ensuring the 3D voxels capture the general human shape), while the other supervises the high-resolution implicit surface function. Interestingly, the 3D decoder used to supervise the voxels is only needed during training; at inference, the model directly interpolates from the latent voxel features.

3. Experimental Setup

The model is evaluated on the DeepHuman dataset, which consists of 5,436 training and 1,359 test meshes. This dataset is 10x larger than the private commercial dataset used in the original PIFu paper and is significantly more diverse, featuring complex poses, varied clothing, and free camera rotations.

4. Results and Discussion

Geo-PIFu demonstrates substantial quantitative and qualitative improvements over existing methods.

Qualitative comparison on the "invisible side" showing Geo-PIFu's superior ability to reconstruct occluded regions compared to DeepHuman and PIFu.

4.1 Quantitative Performance

  • Global Accuracy: Geo-PIFu achieved a 42.7% reduction in Chamfer Distance and Point-to-Surface Distance (PSD) compared to PIFu.
  • Local Detail: Normal estimation errors were reduced by 19.4%, indicating superior capture of fine fabric details.
  • Efficiency: The model is 12x smaller in parameter count than PIFuHD while maintaining higher performance on the DeepHuman benchmark.

4.2 Qualitative Analysis

As shown in Figure 2, Geo-PIFu produces much cleaner reconstructions, particularly for the "invisible" back-side of subjects and complex extremities like hands and feet. The latent voxels act as a regularizer, preventing the model from "hallucinating" impossible body parts in occluded regions and resolving depth ambiguities along camera rays.

5. Conclusion

Geo-PIFu successfully fuses 2D and 3D representations to overcome the limitations of purely pixel-aligned methods. By incorporating a geometry-aligned branch, the model achieves unprecedented structural consistency and detail in single-view human reconstruction. Future work may explore combining this 3D awareness with high-resolution patch-based processing to further enhance reconstruction fidelity.

Summary updated

Abstract

We propose Geo-PIFu, a method to recover a 3D mesh from a monocular color image of a clothed person. Our method is based on a deep implicit function-based representation to learn latent voxel features using a structure-aware 3D U-Net, to constrain the model in two ways: first, to resolve feature ambiguities in query point encoding, second, to serve as a coarse human shape proxy to regularize the high-resolution mesh and encourage global shape regularity. We show that, by both encoding query points and constraining global shape using latent voxel features, the reconstruction we obtain for clothed human meshes exhibits less shape distortion and improved surface details compared to competing methods. We evaluate Geo-PIFu on a recent human mesh public dataset that is 10x larger than the private commercial dataset used in PIFu and previous derivative work. On average, we exceed the state of the art by 42.7% reduction in Chamfer and Point-to-Surface Distances, and 19.4% reduction in normal estimation errors.
BibTeX
@inproceedings{he2020geo,
  author = {Tong He and John P. Collomosse and Hailin Jin and Stefano Soatto},
  title = {{Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=sDzkdrfXYsd},
}
Figure