Humanref: Single image to 3d human generation via reference-guided diffusion

Jingbo Zhang, Xiaoyu Li, Qi Zhang, Yan-Pei Cao, Ying Shan, Jing Liao

Introduction

Generating high-fidelity 3D clothed humans from a single 2D image is a significant challenge in computer vision, often referred to as a "hallucination" problem. The system must accurately predict the appearance of unseen parts of the body—such as the back and sides—while maintaining perfect fidelity to the visible front view. Existing methods typically fall into two categories: reconstruction-based approaches, which often produce blurry textures due to training data limitations, and optimization-based methods using Score Distillation Sampling (SDS). While SDS leverages the generative power of diffusion models, it often relies on low-dimensional text prompts that fail to capture intricate clothing details like specific plaid patterns or unique logos.

Overview of the HumanRef pipeline, transforming a single 2D input image into a high-fidelity 3D human model with consistent textures and geometry.

HumanRef addresses these limitations by introducing a novel framework that combines the generative capabilities of diffusion models with pixel-perfect reference guidance. By ensuring view consistency and texture preservation, HumanRef generates 3D models that are both geometrically sound and visually faithful to the source image.

Methodology

The HumanRef framework represents the 3D human using a Hash-Encoded Signed Distance Field (SDF), optimized through a coarse-to-fine process. The core innovation lies in two primary technical mechanisms:

Reference-Guided Score Distillation Sampling (Ref-SDS)

Standard SDS relies heavily on text prompts, which lack the spatial granularity needed for complex clothing. HumanRef introduces Ref-SDS, which injects features from the original reference image directly into the diffusion model's U-Net attention layers. During the denoising process, the model performs "ref-self-attention," comparing features of the generated novel views against the reference image. This process forces the model to maintain texture consistency across all angles, effectively wrapping the 2D texture around the 3D shape.

Region-Aware Attention

To prevent semantic errors, such as a jacket's texture appearing on the pants (feature bleeding), HumanRef employs region-aware attention. By using human parsing to segment the body into distinct regions (e.g., head, torso, limbs), the model applies masks to the attention matrix. This ensures that the "head" in a generated view only draws guidance from the "head" in the reference image, significantly improving the semantic reasonableness of the output.

Ablation study demonstrating the impact of Reference Guidance. The top row shows results with guidance maintaining texture consistency, while the bottom row shows results without, leading to loss of detail and identity.

Experimental Results

HumanRef was evaluated using high-quality images from the PHORHUM, ICON, and ECON datasets, as well as various "in-the-wild" photos. The framework significantly outperforms state-of-the-art baselines like PIFu, PaMIR, and TeCH in both quantitative and qualitative assessments.

Quantitative Performance

Quantitative results indicate that HumanRef achieves superior scores across key metrics. Specifically, it reached an LPIPS score of 0.032 (compared to TeCH's 0.044), suggesting higher perceptual reconstruction quality. Additionally, its CLIP Score of 90.0% (vs. 85.3% for TeCH) demonstrates stronger semantic similarity to the input image.

Qualitative Comparison

Visual analysis shows that HumanRef maintains sharp, consistent textures—such as complex plaid patterns—where competing methods often produce blurry or inconsistent results. Furthermore, the geometry generated by HumanRef is cleaner, avoiding the "melted" artifacts common in other SDS-based approaches, particularly around the feet and extremities.

Limitations and Future Work

Despite its performance, HumanRef faces two primary challenges. First is the "Janus Problem," where the model may occasionally generate a second face on the back of the head due to a lack of explicit back-view constraints. Second, the quality of the 3D generation is heavily dependent on the accuracy of the initial SMPL-X pose estimation. Future research will focus on incorporating view-dependent constraints to eliminate multi-face artifacts and improving robustness to varied human poses.

Summary updated

Abstract

Generating a 3D human model from a single reference image is challenging because it requires inferring textures and geometries in invisible views while maintaining consistency with the reference image. Previous methods utilizing 3D generative models are limited by the availability of 3D training data. Optimization-based methods that lift text-to-image diffusion models to 3D generation often fail to preserve the texture details of the reference image resulting in inconsistent appearances in different views. In this paper we propose HumanRef a 3D human generation framework from a single-view input. To ensure the generated 3D model is photorealistic and consistent with the input image HumanRef introduces a novel method called reference-guided score distillation sampling (Ref-SDS) which effectively incorporates image guidance into the generation process. Furthermore we introduce region-aware attention to Ref-SDS ensuring accurate correspondence between different body regions. Experimental results demonstrate that HumanRef outperforms state-of-the-art methods in generating 3D clothed humans with fine geometry photorealistic textures and view-consistent appearances. Code and model are available at https://eckertzhang.github.io/HumanRef.github.io/.
BibTeX
@inproceedings{zhang2024humanref,
  author = {Jingbo Zhang and Xiaoyu Li and Qi Zhang and Yan-Pei Cao and Ying Shan and Jing Liao},
  title = {{Humanref: Single image to 3d human generation via reference-guided diffusion}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=2aQ3aFFEZF},
}
Figure