StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision

Yang Hong, Juyong Zhang, Boyi Jiang, Yudong Guo, Ligang Liu, Hujun Bao

Introduction

The creation of high-fidelity 3D digital doubles of clothed humans is a cornerstone for emerging technologies in AR/VR, virtual try-ons, and telepresence. While high-end multi-camera rigs provide high-quality results, their cost and complexity limit accessibility for average consumers. Recent advancements in implicit function representations, such as the Pixel-Aligned Implicit Function (PIFu), have enabled 3D reconstruction from a single image. However, single-view methods inherently suffer from depth ambiguity. Without true 3D spatial information, these models often produce "broken limbs"—where extremities are detached from the torso—and struggle with anatomical consistency when viewed from non-frontal angles.

StereoPIFu addresses these challenges by integrating stereo vision constraints with implicit functions. By leveraging binocular images and a novel "voxel-aligned" feature set, the system produces high-fidelity, depth-aware reconstructions that maintain correct topology even in complex poses.

Methodology

StereoPIFu reformulates 3D human reconstruction as a stereo-matching problem integrated with occupancy prediction. The pipeline consists of several key components designed to bridge the gap between 2D images and 3D space.

Stereo Vision Backbone and Cost Volume

The system processes a pair of rectified stereo images through a stereo-matching network (AANet+). This network constructs a 3D cost volume that encodes the correlation between pixels in the left and right images. This volume serves as the foundation for capturing precise depth information through disparity, providing the geometric constraints missing in monocular approaches.

Voxel-Aligned Features

Unlike standard PIFu, which relies on 2D pixel-aligned features, StereoPIFu extracts voxel-aligned features directly from the 3D cost volume. This approach allows the network to understand the human subject's actual position in camera space, rather than just their projection on a 2D plane. This spatial awareness is vital for multi-person performance capture and accurate world-space positioning.

Relative Z-Offset and Depth Awareness

To refine surface details, the authors introduce a "relative z-offset." This variable represents the distance between a 3D query point and the predicted surface depth at that specific pixel. By providing this offset as an input, the occupancy network can focus on fine-grained geometric details relative to a known surface, rather than attempting to learn absolute coordinates from scratch. This acts as a "bridge" that simplifies the learning task for the occupancy MLP.

Handling Occlusions

To prevent "geometry copy" artifacts—where front-side details are erroneously mirrored onto the back—a specialized transformation function $\psi_t(x)$ is used to normalize depth offsets. This ensures that the model remains robust even when body parts (like hands) occlude the torso, maintaining a clear distinction between front and back geometry.

Experimental Setup

Training a stereo-aware model requires significant data diversity. The authors utilized a combination of synthetic and real-world data:

  • Synthetic Data: 274 high-quality rigged human models from AXYZ and RenderPeople, animated with Mixamo to generate over 68,000 unique meshes. These were rendered as binocular pairs with varied lighting and camera parameters.
  • Benchmarks: Evaluation was conducted on the AXYZ Pose dataset (100 complex poses) and the BUFF dataset (real-world 3D scans of humans in motion).
  • Real-world Testing: The model's generalization was verified using images captured with a commercial ZED stereo camera to prove its utility in consumer-grade setups.

Results

StereoPIFu demonstrates a significant performance leap over state-of-the-art methods like DeepHuman and PIFuHD.

Quantitative Performance

On the BUFF dataset, StereoPIFu achieved a 74.1% reduction in Point-to-Surface (P2S) error and a 75.1% reduction in Chamfer distance compared to PIFuHD. These metrics underscore the precision gained by utilizing stereo constraints over simple multi-view feature pooling.

Qualitative Improvements

The model effectively eliminates the "broken limb" artifacts common in single-view approaches. Even in extreme poses, such as a person bending over or reaching forward, StereoPIFu maintains correct anatomical topology. Furthermore, because the model operates in camera space, the resulting 3D models are correctly positioned relative to the camera, avoiding the scale and position drift typical of monocular methods.

Discussion and Limitations

The primary insight of StereoPIFu is that stereo constraints provide a more rigorous geometric foundation than simple multi-view feature pooling. By utilizing cost volumes, the model "understands" depth through disparity rather than just averaging features.

However, the method is not without limitations. It is highly dependent on accurate camera calibration; poor calibration leads to significant degradation in reconstruction quality. Additionally, while the model generates plausible geometry for occluded regions (the back of the subject), these areas cannot be guaranteed to be accurate as they are not seen by the cameras. Future work may involve extending this framework to Multi-View Stereo (MVS) setups with three or more cameras to further minimize occlusions and enhance completeness.

Conclusion

StereoPIFu represents a significant step forward in making high-quality 3D human digitization more accessible. By bridging the gap between traditional stereo vision and modern implicit representations, it provides a robust solution for depth-aware human reconstruction in real-world environments.

Summary updated

BibTeX
@inproceedings{hong2021stereopifu,
  author = {Yang Hong and Juyong Zhang and Boyi Jiang and Yudong Guo and Ligang Liu and Hujun Bao},
  title = {{StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00060},
}
Figure