HDhuman: High-Quality Human Novel-View Rendering From Sparse Views

Tiansong Zhou, Jing Huang, Tao Yu, Ruizhi Shao, Kun Li

TL;DR

This paper introduces HDhuman, a novel method for generating high-quality, 2K resolution novel-view renderings of humans from a very limited number of input images (as few as 6). It achieves this by first reconstructing highly detailed human geometry, including fine details like clothing folds, and then using this geometry to guide a visibility-aware feature integration process for rendering, effectively solving occlusion problems.

What Problem Does This Paper Address?

Realistic free-viewpoint rendering of humans is a key goal for applications like AR/VR. While methods like Neural Radiance Fields (NeRF) excel with many input views, their performance degrades significantly with sparse views. Existing approaches that try to address sparse-view human rendering often struggle to capture high-frequency geometric details, especially for clothing with complex textures or folds. This limitation leads to blurry or artifact-ridden renderings. Furthermore, many previous methods require training a separate network for each individual human, which is computationally expensive and limits their generalizability.

Methods

HDhuman employs a two-stage pipeline:

  1. High-Quality Human Reconstruction:
    • Pixel-Aligned Spatial Transformer: Instead of simple averaging of features from multiple views (as in PIFu), this novel transformer calculates correlations between input views. This allows it to preserve fine geometric details, such as clothing folds, that would otherwise be lost.
  2. The reconstruction network is designed to be general, not requiring a human template like SMPL, leading to fully pixel-aligned reconstructions.
  3. Geometry-Guided Novel-View Rendering:
    • Pixel-wise Visibility Reasoning: Based on the reconstructed geometry, this module determines which source views are visible for each pixel in the novel view. This is crucial for handling occlusions caused by sparse inputs.
  4. Feature Integration: Only features from visible source views are integrated into the novel view pixel. This process is guided by the reconstructed geometry, preventing artifacts from invisible or occluded regions.
  5. The method uses a convolutional render-net to produce the final high-resolution (2K) novel-view images.

Datasets

  • Twindom Dataset: A large-scale dataset of 1700 high-quality textured human meshes was collected and used for training and evaluation of the reconstruction and rendering networks. This dataset features a wide range of clothing, poses, and shapes.
  • Real-World Multi-View Videos: A custom dataset was created capturing 5 dynamic human videos with 24 synchronized cameras. This dataset specifically includes human performers with complex texture patterns and loose clothing, posing a greater challenge than datasets with uniform textures.

Key Insights and Ideas

  • Geometry is Key for Detail: Recovering fine geometric details, particularly clothing folds, is essential for high-quality rendering from sparse views, especially for complex human subjects.
  • Correlation, Not Just Averaging: Simple averaging of multi-view features is insufficient for capturing subtle geometry. Learning correlations between views via a spatial transformer is critical.
  • Visibility Reasoning Solves Occlusion: Explicitly reasoning about pixel visibility using reconstructed geometry is a robust way to handle occlusions in sparse-view rendering, a common problem in traditional methods.
  • Generalization is Achievable: The proposed framework demonstrates strong generalization capabilities, performing well on novel subjects without requiring fine-tuning.

Key Results

  • Quantitative Improvements: HDhuman significantly outperforms prior methods like Neural Body, Soft3D, and SA-NeRF across various metrics (LPIPS, SSIM, PSNR) on both synthetic (Twindom) and real-world datasets. For example, on the Twindom dataset, the average LPIPS score for HDhuman is 0.165, compared to 0.293 for Neural Body.
  • High-Quality Reconstruction: The method achieves highly detailed reconstructions, preserving intricate details like clothing folds, even from as few as 6 input views.
  • Robust Rendering: HDhuman produces high-quality 2K resolution novel-view renderings, effectively handling challenging cases with complex textures and loose clothing, as demonstrated in qualitative results.
  • Efficiency with Sparse Views: The method shows minimal performance degradation when the number of input views is reduced, indicating its effectiveness with highly sparse inputs.

Limitations and Future Work

  • Reconstruction Dependency: The rendering quality is directly dependent on the accuracy of the reconstructed geometry. Failures in reconstruction lead to rendering artifacts.
  • Indifferentiable Geometry: The current geometry-guided feature integration is not differentiable with respect to the reconstructed geometry. This prevents refining the geometry during fine-tuning, which could otherwise help resolve reconstruction errors.
  • Future Work Suggestion: The authors suggest exploring differentiable geometry representations (e.g., volumes) to enable differentiable geometry-guided visibility reasoning and potentially refine geometry during fine-tuning. This could lead to even more robust and higher-quality rendering.

Summary updated

Abstract

In this paper, we aim to address the challenge of novel view rendering of human performers that wear clothes with complex texture patterns using a sparse set of camera views. Although some recent works have achieved remarkable rendering quality on humans with relatively uniform textures using sparse views, the rendering quality remains limited when dealing with complex texture patterns as they are unable to recover the high-frequency geometry details that are observed in the input views. To this end, we propose HDhuman, which uses a human reconstruction network with a pixel-aligned spatial transformer and a rendering network with geometry-guided pixel-wise feature integration to achieve high-quality human reconstruction and rendering. The designed pixel-aligned spatial transformer calculates the correlations between the input views and generates human reconstruction results with high-frequency details. Based on the surface reconstruction results, the geometry-guided pixel-wise visibility reasoning provides guidance for multi-view feature integration, enabling the rendering network to render high-quality images at 2k resolution on novel views. Unlike previous neural rendering works that always need to train or fine-tune an independent network for a different scene, our method is a general framework that is able to generalize to novel subjects. Experiments show that our approach outperforms all the prior generic or specific methods on both synthetic data and real-world data. Source code and test data will be made publicly available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/HDhuman/index.html.
BibTeX
@article{zhou2024hdhuman,
  author = {Tiansong Zhou and Jing Huang and Tao Yu and Ruizhi Shao and Kun Li},
  title = {{HDhuman: High-Quality Human Novel-View Rendering From Sparse Views}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2024},
  doi = {10.1109/tvcg.2023.3290543},
}
Figure