PIDiffu: Pixel-aligned Diffusion Model for High-Fidelity Clothed Human Reconstruction

Jungeun Lee, Sanghun Kim, Hansol Lee, Tserendorj Adiya, Hwasup Lim

Introduction

Reconstructing high-fidelity 3D clothed humans from a single image is a cornerstone task for applications ranging from virtual reality to e-commerce. While Pixel-Aligned Implicit Functions (PIFu) have set a strong baseline, they often struggle with structural consistency and generalization to complex real-world scenarios. PIDiffu addresses these limitations by reimagining the reconstruction process as a line-wise diffusion task.

The Challenge of Single-Image Reconstruction

Traditional methods like PIFu predict occupancy or Signed Distance Functions (SDF) on a point-by-point basis. This independent prediction often leads to "depth ambiguity," manifesting as floating limbs, elongated torsos, or duplicated geometry. Furthermore, these models frequently fail when encountering "in-the-wild" images with unusual poses or intricate clothing patterns, as the conditioning mechanisms (like FiLM) can over-modulate and lead to unstable results.

The PIDiffu Approach: Line-wise Diffusion

PIDiffu shifts the paradigm from point-wise estimation to line-wise distribution estimation along camera rays.

Ray-based Diffusion

Instead of evaluating points in isolation, PIDiffu uses a diffusion model to generate the SDF for an entire ray simultaneously. This 1D approach captures the structural context necessary to ensure geometric consistency without the prohibitive computational cost of full 3D voxel grids.

Feature-wise Linear Attention (FiLA)

To improve robustness, the authors introduce FiLA. Unlike standard modulation techniques that can drastically alter feature maps, FiLA uses an attention-based residual mechanism. This allows the model to integrate image features while staying anchored to the robust geometric priors learned during training, preventing the "explosion" of artifacts in challenging cases.

Input image featuring complex clothing and accessories.

Experimental Results

The model was trained on the THuman2.0 dataset and evaluated on benchmarks like BUFF and SSHQ.

Quantitative Performance

PIDiffu consistently outperforms state-of-the-art methods. On the BUFF dataset, it achieved a Chamfer Distance (CD) of 2.861, surpassing IntegratedPIFu's 2.968.

Qualitative Superiority

Visually, PIDiffu produces significantly cleaner meshes. It excels at reconstructing accessories like bags and hats—elements that typically cause baseline models to fail. By maintaining a consistent noise set during diffusion inference, the model ensures a smooth and continuous 3D surface.

PIDiffu's high-fidelity 3D reconstruction result.

Conclusion and Future Work

PIDiffu demonstrates that modeling 1D distributions along rays is a "sweet spot" for 3D human reconstruction, balancing efficiency with structural awareness. While it significantly reduces artifacts, challenges remain regarding extreme self-occlusion and dataset-induced demographic biases. Future work will focus on integrating parametric body models and expanding training diversity to further enhance generalization.

Summary updated

Abstract

This paper presents the Pixel-aligned Diffusion Model (PIDiffu), a new framework for reconstructing high-fidelity clothed 3D human models from a single image. While existing PIFu variants have made significant advances using more complicated 2D and 3D feature extractions, these methods still suffer from floating artifacts and body part du-plication due to their reliance on point-wise occupancy field estimations. PIDiffu employs a diffusion-based strategy for line-wise estimation along the ray direction, conditioned by pixel-aligned features with a guided attention. This approach improves the local details and structural accuracy of the reconstructed body shape and is robust to unfamiliar and complex image features. Moreover, PIDiffu can be easily integrated with existing PIFu-based methods to leverage their advantages. The paper demonstrates that PIDiffu outperforms state-of-the-art methods that do not rely on parametric 3D body models. Especially, our method is superior in handling ’in-the-wild’ images, such as those with complex patterned clothes unseen in the training data.
BibTeX
@inproceedings{lee2024pidiffu,
  author = {Jungeun Lee and Sanghun Kim and Hansol Lee and Tserendorj Adiya and Hwasup Lim},
  title = {{PIDiffu: Pixel-aligned Diffusion Model for High-Fidelity Clothed Human Reconstruction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00509},
}
Figure