Intrinsic Hand Avatar: Illumination-Aware Hand Appearance and Shape Reconstruction From Monocular RGB Video

Pratik Kalshetti, Parag ChaudhuriWACV 2024

TL;DR

This paper introduces a novel method for reconstructing a personalized, "intrinsic" hand avatar from a single, monocular RGB video. Unlike previous approaches that bake lighting into appearance, this method disentangles intrinsic hand properties (shape, material) from environment lighting, enabling realistic self-shadows and novel relighting capabilities.

What Problem Does This Paper Address?

Creating realistic and personalized digital avatars is crucial for immersive virtual and augmented reality experiences. While significant progress has been made in reconstructing human avatars and faces, hands present unique challenges due to their complex articulation, frequent self-occlusion, and critical role in interaction. Existing hand avatar reconstruction methods often rely on multi-view setups, struggle to disentangle lighting from appearance (leading to "baked-in" shadows and illumination), and require extensive training. This paper aims to address these limitations by enabling the reconstruction of a high-fidelity, intrinsic hand avatar from a simple monocular video, even under arbitrary poses and real-world lighting conditions, in a matter of minutes.

Methods

The core of the proposed method is a differentiable rendering framework that jointly optimizes hand geometry, appearance, and environment lighting.

  • Geometry: The method starts with the widely-used MANO hand model, which provides a parametric representation of hand shape and pose. To capture finer geometric details, they extend MANO by linearly subdividing the template mesh and introducing per-vertex offsets. This allows for more detailed and user-specific geometry.
  • Appearance: The hand's material reflectance is modeled using Disney's principled BRDF, parameterized by diffuse albedo ($k_d$), roughness ($r$), metalness ($m$), and a tangent space normal map ($n$). This physically-based model allows for realistic rendering of various skin appearances.
  • Lighting: Environment lighting is represented by a high dynamic range (HDR) environment probe, which captures incident radiance from all directions. This is crucial for disentangling lighting from the intrinsic appearance.
  • Differentiable Rendering: A deferred rendering pipeline is employed, incorporating Monte Carlo path tracing to estimate outgoing radiance. This allows for explicit handling of self-shadows, which are vital for realistic hand rendering. The entire pipeline is differentiable, enabling gradient-based optimization.
  • Optimization: The method uses an analysis-by-synthesis approach. Given a monocular RGB video, it jointly optimizes the geometry (MANO parameters and offsets), material parameters, and the HDR light probe. This is achieved by minimizing a total loss function that includes data terms (pixel-wise and mask loss), geometry regularization (Laplacian), material regularization (smoothness priors), and a perceptual loss (LPIPS). Lighting is regularized using a monochrome image loss.

Datasets

  • InterHand2.6M dataset [34]: Used for quantitative evaluation and comparison with state-of-the-art methods. Images are centered, scaled, and cropped.
  • Custom Captured Dataset: The authors also captured their own dataset using a calibrated RGB camera, featuring 7 different users and approximately 1000 frames. This dataset is designed to mimic real-world AR/VR scenarios with challenging lighting and diverse hand shapes.

Key Insights and Ideas

  • Disentangling Intrinsic Properties: The most significant insight is the successful disentanglement of intrinsic hand appearance (material) from environment lighting. This is key to achieving realistic self-shadows and enabling novel relighting applications, a common limitation in prior work.
  • Efficient Reconstruction: Despite using physically-based rendering and Monte Carlo path tracing, the method achieves reconstruction within minutes, making it practical for real-time applications.
  • Leveraging Existing Models with Extensions: The paper effectively builds upon established models like MANO and integrates them with novel extensions (mesh subdivision, per-vertex offsets) to achieve higher fidelity geometry.
  • Analysis-by-Synthesis for Monocular Video: The use of an analysis-by-synthesis approach with a differentiable renderer is powerful for reconstructing detailed avatars from a single, uncalibrated video stream.

Key Results

  • Quantitative Performance: The method outperforms existing state-of-the-art methods (HARP, HandAvatar, SelfRecon, HumanNeRF) on standard metrics like PSNR, SSIM, and LPIPS across various datasets and frame rates. For example, on the InterHand2.6M 30fps dataset, their method achieved a PSNR of 31.179, SSIM of 0.936, and LPIPS of 0.061.
  • Qualitative Improvements: The reconstructed avatars exhibit sharper texture details, more accurate self-shadows, and better handling of lighting compared to prior methods. Figure 5 clearly demonstrates the superior shadow-aware capability.
  • Generalization: The method shows good generalization to out-of-distribution appearances, as demonstrated in Figure 9, suggesting its suitability for diverse users in AR/VR.
  • Speed: Reconstruction is completed within minutes (e.g., 10 minutes for 100 frames at 1024x1024 resolution on an RTX 3080 Ti).

Limitations and Future Work

  • Initialization Sensitivity: The method is sensitive to the quality of the initial mesh parameters and the segmentation mask. A good initialization is crucial for successful optimization.
  • Occlusion Handling: The method assumes the hand is clearly visible and cannot handle occlusions caused by other objects. Regions of the hand never visible in the video will retain their initialized values.
  • Static Lighting: While the method handles dynamic hand poses, it assumes static lighting throughout the input video.
  • Material Limitations: The current material model has limitations, such as clamping metalness to low values, which prevents modeling of very shiny fingernails.
  • Future Work: The authors suggest improving the material model with subsurface scattering, incorporating bulges and wrinkles into the geometry deformation, and modeling fine hair on the back of the hand for enhanced realism. They also highlight the ethical concern of fingerprint privacy and mention down-sampling images to mitigate this.

Summary updated

BibTeX
@inproceedings{kalshetti2024intrinsic,
  author = {Pratik Kalshetti and Parag Chaudhuri},
  title = {{Intrinsic Hand Avatar: Illumination-Aware Hand Appearance and Shape Reconstruction From Monocular RGB Video}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure