Introduction
Generating high-fidelity 3D clothed humans from a single 2D image is a significant challenge in computer vision, often referred to as a "hallucination" problem. The system must accurately predict the appearance of unseen parts of the body—such as the back and sides—while maintaining perfect fidelity to the visible front view. Existing methods typically fall into two categories: reconstruction-based approaches, which often produce blurry textures due to training data limitations, and optimization-based methods using Score Distillation Sampling (SDS). While SDS leverages the generative power of diffusion models, it often relies on low-dimensional text prompts that fail to capture intricate clothing details like specific plaid patterns or unique logos.
HumanRef addresses these limitations by introducing a novel framework that combines the generative capabilities of diffusion models with pixel-perfect reference guidance. By ensuring view consistency and texture preservation, HumanRef generates 3D models that are both geometrically sound and visually faithful to the source image.
Methodology
The HumanRef framework represents the 3D human using a Hash-Encoded Signed Distance Field (SDF), optimized through a coarse-to-fine process. The core innovation lies in two primary technical mechanisms:
Reference-Guided Score Distillation Sampling (Ref-SDS)
Standard SDS relies heavily on text prompts, which lack the spatial granularity needed for complex clothing. HumanRef introduces Ref-SDS, which injects features from the original reference image directly into the diffusion model's U-Net attention layers. During the denoising process, the model performs "ref-self-attention," comparing features of the generated novel views against the reference image. This process forces the model to maintain texture consistency across all angles, effectively wrapping the 2D texture around the 3D shape.
Region-Aware Attention
To prevent semantic errors, such as a jacket's texture appearing on the pants (feature bleeding), HumanRef employs region-aware attention. By using human parsing to segment the body into distinct regions (e.g., head, torso, limbs), the model applies masks to the attention matrix. This ensures that the "head" in a generated view only draws guidance from the "head" in the reference image, significantly improving the semantic reasonableness of the output.
Experimental Results
HumanRef was evaluated using high-quality images from the PHORHUM, ICON, and ECON datasets, as well as various "in-the-wild" photos. The framework significantly outperforms state-of-the-art baselines like PIFu, PaMIR, and TeCH in both quantitative and qualitative assessments.
Quantitative Performance
Quantitative results indicate that HumanRef achieves superior scores across key metrics. Specifically, it reached an LPIPS score of 0.032 (compared to TeCH's 0.044), suggesting higher perceptual reconstruction quality. Additionally, its CLIP Score of 90.0% (vs. 85.3% for TeCH) demonstrates stronger semantic similarity to the input image.
Qualitative Comparison
Visual analysis shows that HumanRef maintains sharp, consistent textures—such as complex plaid patterns—where competing methods often produce blurry or inconsistent results. Furthermore, the geometry generated by HumanRef is cleaner, avoiding the "melted" artifacts common in other SDS-based approaches, particularly around the feet and extremities.
Limitations and Future Work
Despite its performance, HumanRef faces two primary challenges. First is the "Janus Problem," where the model may occasionally generate a second face on the back of the head due to a lack of explicit back-view constraints. Second, the quality of the 3D generation is heavily dependent on the accuracy of the initial SMPL-X pose estimation. Future research will focus on incorporating view-dependent constraints to eliminate multi-face artifacts and improving robustness to varied human poses.
Summary updated