Abstract
Hand image synthesis and pose estimation from RGB images are both highly challenging tasks due to the large discrepancy between factors of variation ranging from image background content to camera viewpoint. To better analyze these factors of variation, we propose the use of disentangled representations and a disentangled variational autoencoder (dVAE) that allows for specific sampling and inference of these factors. The derived objective from the variational lower bound as well as the proposed training strategy are highly flexible, allowing us to handle cross-modal encoders and decoders as well as semi-supervised learning scenarios. Experiments show that our dVAE can synthesize highly realistic images of the hand specifiable by both pose and image background content and also estimate 3D hand poses from RGB images with accuracy competitive with state-of-the-art on two public benchmarks.
Introduction
Estimating the 3D pose of a human hand from a standard RGB image is a classic computer vision challenge. Unlike depth sensors, RGB images lack explicit spatial information and are often plagued by self-occlusion and cluttered backgrounds. Traditional deep learning models often treat the internal representation of a hand as a "black box," offering little control over specific factors like pose or environment during generation. Furthermore, the scarcity of high-quality 3D labels for real-world RGB images often forces a reliance on synthetic data that may not generalize well.
To address these challenges, researchers have developed a disentangled Variational Autoencoder (dVAE). This model separates the complex factors of a hand image—such as 3D pose, camera viewpoint, and background—into independent latent variables. This structured approach enables both high-accuracy 3D pose estimation and the synthesis of realistic hand images with user-controlled attributes.
The dVAE Framework
The core innovation of the dVAE lies in its structured latent space. Instead of a single monolithic vector, the latent space is partitioned into distinct sub-spaces:
- Canonical Pose: Represents the intrinsic shape and configuration of the hand.
- Viewpoint: Captures the rotation and orientation relative to the camera.
- Nuisance Variables ($z_u$): Captures "everything else," including background content, lighting, and noise.
Training Strategy
The model employs a two-step training process. First, a disentangling step maps specific labels (like 3D coordinates) to their respective sub-spaces. Second, an embedding step teaches the model to map raw RGB images into this structured latent space. This cross-modal flexibility allows the dVAE to bridge the gap between perfectly labeled synthetic data and unlabeled real-world photos. By explicitly modeling the "nuisance" background factors in $z_u$, the model prevents background noise from corrupting the pose representation, leading to more robust estimation.
Experimental Results
The dVAE was evaluated on the Stereo Hand Pose Tracking Benchmark (STB) and the more challenging Rendered Hand Pose Dataset (RHD).
Pose Estimation
On the RHD dataset, the dVAE achieved a Mean End-Point Error (EPE) of 19.95mm, a significant improvement over previous baselines like Zimmermann et al. (30.42mm). The use of weak labels (e.g., viewpoint only) further improved accuracy by up to 7.5%, demonstrating the benefits of the semi-supervised approach.
Image Synthesis and Pose Transfer
The disentangled nature of the latent space allows for "one-to-many" synthesis. A single pose can be rendered against multiple backgrounds by simply swapping the $z_u$ variable. The model also supports "latent space walks," where the pose variable is smoothly interpolated while the background remains static, producing realistic animations.
Conclusion
The dVAE framework demonstrates that disentangling factors of variation is key to both robust inference and controllable synthesis in complex domains like hand tracking. While current assumptions of factor independence could be further refined to account for physical constraints, the model's ability to leverage cross-modal data and weak supervision sets a new standard for RGB-based hand analysis.
Summary updated