TL;DR
This paper introduces HDhuman, a novel method for generating high-quality, 2K resolution novel-view renderings of humans from a very limited number of input images (as few as 6). It achieves this by first reconstructing highly detailed human geometry, including fine details like clothing folds, and then using this geometry to guide a visibility-aware feature integration process for rendering, effectively solving occlusion problems.
What Problem Does This Paper Address?
Realistic free-viewpoint rendering of humans is a key goal for applications like AR/VR. While methods like Neural Radiance Fields (NeRF) excel with many input views, their performance degrades significantly with sparse views. Existing approaches that try to address sparse-view human rendering often struggle to capture high-frequency geometric details, especially for clothing with complex textures or folds. This limitation leads to blurry or artifact-ridden renderings. Furthermore, many previous methods require training a separate network for each individual human, which is computationally expensive and limits their generalizability.
Methods
HDhuman employs a two-stage pipeline:
- High-Quality Human Reconstruction:
- Pixel-Aligned Spatial Transformer: Instead of simple averaging of features from multiple views (as in PIFu), this novel transformer calculates correlations between input views. This allows it to preserve fine geometric details, such as clothing folds, that would otherwise be lost.
- The reconstruction network is designed to be general, not requiring a human template like SMPL, leading to fully pixel-aligned reconstructions.
- Geometry-Guided Novel-View Rendering:
- Pixel-wise Visibility Reasoning: Based on the reconstructed geometry, this module determines which source views are visible for each pixel in the novel view. This is crucial for handling occlusions caused by sparse inputs.
- Feature Integration: Only features from visible source views are integrated into the novel view pixel. This process is guided by the reconstructed geometry, preventing artifacts from invisible or occluded regions.
- The method uses a convolutional render-net to produce the final high-resolution (2K) novel-view images.
Datasets
- Twindom Dataset: A large-scale dataset of 1700 high-quality textured human meshes was collected and used for training and evaluation of the reconstruction and rendering networks. This dataset features a wide range of clothing, poses, and shapes.
- Real-World Multi-View Videos: A custom dataset was created capturing 5 dynamic human videos with 24 synchronized cameras. This dataset specifically includes human performers with complex texture patterns and loose clothing, posing a greater challenge than datasets with uniform textures.
Key Insights and Ideas
- Geometry is Key for Detail: Recovering fine geometric details, particularly clothing folds, is essential for high-quality rendering from sparse views, especially for complex human subjects.
- Correlation, Not Just Averaging: Simple averaging of multi-view features is insufficient for capturing subtle geometry. Learning correlations between views via a spatial transformer is critical.
- Visibility Reasoning Solves Occlusion: Explicitly reasoning about pixel visibility using reconstructed geometry is a robust way to handle occlusions in sparse-view rendering, a common problem in traditional methods.
- Generalization is Achievable: The proposed framework demonstrates strong generalization capabilities, performing well on novel subjects without requiring fine-tuning.
Key Results
- Quantitative Improvements: HDhuman significantly outperforms prior methods like Neural Body, Soft3D, and SA-NeRF across various metrics (LPIPS, SSIM, PSNR) on both synthetic (Twindom) and real-world datasets. For example, on the Twindom dataset, the average LPIPS score for HDhuman is 0.165, compared to 0.293 for Neural Body.
- High-Quality Reconstruction: The method achieves highly detailed reconstructions, preserving intricate details like clothing folds, even from as few as 6 input views.
- Robust Rendering: HDhuman produces high-quality 2K resolution novel-view renderings, effectively handling challenging cases with complex textures and loose clothing, as demonstrated in qualitative results.
- Efficiency with Sparse Views: The method shows minimal performance degradation when the number of input views is reduced, indicating its effectiveness with highly sparse inputs.
Limitations and Future Work
- Reconstruction Dependency: The rendering quality is directly dependent on the accuracy of the reconstructed geometry. Failures in reconstruction lead to rendering artifacts.
- Indifferentiable Geometry: The current geometry-guided feature integration is not differentiable with respect to the reconstructed geometry. This prevents refining the geometry during fine-tuning, which could otherwise help resolve reconstruction errors.
- Future Work Suggestion: The authors suggest exploring differentiable geometry representations (e.g., volumes) to enable differentiable geometry-guided visibility reasoning and potentially refine geometry during fine-tuning. This could lead to even more robust and higher-quality rendering.
Summary updated