HDhuman: High-quality Human Performance Capture with Sparse Views

Tiansong Zhou, Tao Yu, Ruizhi Shao, Kun LiarXiv 2022 · OpenReview

Abstract

HDhuman is a generalizable framework designed to render high-quality, 2k-resolution images of human performers from a sparse set of 6 to 8 camera views. By integrating a pixel-aligned spatial transformer for detailed 3D reconstruction with a visibility-aware rendering network, the system captures complex clothing textures and loose garments. Unlike existing methods, HDhuman operates in a single feed-forward pass, eliminating the need for time-consuming per-subject fine-tuning.

Introduction

Creating realistic, free-viewpoint renderings of humans is essential for AR/VR applications. However, achieving high fidelity with sparse data—defined as very few camera views—remains a significant challenge. While Neural Radiance Fields (NeRF) produce photorealistic results with dense data, their performance degrades significantly with sparse views. Conversely, models relying on the SMPL body proxy often fail to capture high-frequency details such as clothing folds or loose garments. Furthermore, many state-of-the-art methods are subject-specific, requiring hours of optimization for each new individual. HDhuman addresses these limitations by providing a generalizable model capable of recovering fine geometry and complex textures from sparse views.

Methodology

The HDhuman framework utilizes a two-stage reconstruction-rendering pipeline.

1. Human Reconstruction Network

The first stage builds upon the Pixel-aligned Implicit Function (PIFu) approach by introducing a pixel-aligned spatial transformer. Traditional methods often use simple average pooling to fuse multi-view data, which can lead to loss of detail. In contrast, the spatial transformer calculates correlations between views, allowing the network to attend to the most relevant features for any 3D point. This mechanism preserves high-frequency geometric details, such as wrinkles and folds in clothing.

The Human Reconstruction Network utilizes a pixel-aligned spatial transformer to fuse multi-view features for 3D occupancy prediction.

2. Rendering Network

Following reconstruction, the system employs a rendering network that performs geometry-guided pixel-wise visibility reasoning. For each pixel in a novel view, the network determines which source cameras have an unobstructed view of the corresponding 3D point. By selectively integrating features only from visible source views, the model effectively eliminates blurring and "ghosting" artifacts typically associated with image-based rendering.

The Rendering Network performs visibility reasoning to integrate features from visible source views, guided by the reconstructed geometry.

Datasets and Evaluation

The model was trained and evaluated using two primary datasets:

  • Twindom Dataset: A collection of 1,700 high-quality textured human meshes (1,500 for training, 200 for testing) used for ground-truth geometry and texture.
  • Multi-view Video Dataset: A custom real-world dataset featuring 5 subjects in dynamic motion, captured at 2660 × 2300 resolution with a 24-camera system to test performance under complex lighting and with loose clothing.

Results

HDhuman demonstrates significant improvements over existing baselines:

  • Reconstruction Accuracy: On the Twindom dataset, HDhuman reduced the Chamfer distance error to 0.2341, compared to 0.9873 for "Neural Body."
  • Rendering Quality: The framework consistently outperformed methods like Soft3D, Neural Body, and Neural Texture in terms of SSIM and LPIPS metrics.
  • Robustness: Qualitative evaluations show the model successfully rendering challenging subjects, such as performers in long, patterned dresses, using only 6 views—a scenario where competing methods often fail.

Discussion and Future Work

A key insight of this work is that high-fidelity texture rendering is intrinsically linked to fine-scale geometry reconstruction. By moving beyond simple average pooling to a transformer-based fusion and implementing explicit visibility reasoning, HDhuman achieves superior results in sparse-view scenarios.

However, limitations remain. The final rendering quality is highly dependent on the initial reconstruction; errors in the 3D mesh directly translate to visual artifacts. Future research will explore differentiable geometry representations, allowing the rendering loss to refine the 3D shape during the process. Additionally, while the model is generalizable, further work is needed to enable effective subject-specific fine-tuning that can refine geometric details.

Summary updated

BibTeX
@misc{zhou2022hdhuman,
  author = {Tiansong Zhou and Tao Yu and Ruizhi Shao and Kun Li},
  title = {{HDhuman: High-quality Human Performance Capture with Sparse Views}},
  year = {2022},
  url = {https://openreview.net/forum?id=DO7xRfw8i6X},
}
Figure