Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation

Zhuoran Zhao, Linlin Yang, Pengzhan Sun, Pan Hui, Angela Yao

Abstract

While synthetic data has revolutionized face and body estimation, 3D hand pose estimation has long suffered from a stubborn "synthetic-to-real" domain gap. This research systematically deconstructs this gap, identifying the forearm, image frequency statistics, and object occlusions as the primary culprits. By introducing a high-quality synthesis pipeline and a novel "Decomposition and Composition" strategy, this work demonstrates that models trained on synthetic data can finally match the accuracy of those trained on real-world datasets.

Introduction

In computer vision, synthetic data is highly valued for providing "free" perfect labels. For tasks such as face recognition or human body pose estimation, models trained purely on synthetic data have reached state-of-the-art performance. However, 3D hand pose estimation has remained a notable outlier. Despite advances in photorealistic rendering, models trained on synthetic hands often experience a significant performance drop when applied to real-world imagery. This paper addresses a fundamental question: What exactly constitutes the synthetic-to-real gap for hands? By isolating variables such as lighting, anatomical completeness, and environmental interaction, the authors aim to eliminate the need for expensive, manual 3D hand labeling.

Methodology

The researchers developed a sophisticated data synthesis pipeline and a "Decomposition and Composition" strategy to isolate specific variables contributing to the domain gap.

High-Fidelity Modeling

The pipeline utilizes the NIMBLE hand model, which features a significantly denser mesh (5,990 vertices) compared to the industry-standard MANO model (778 vertices). This increased density allows for more realistic simulations of skin, muscle, and bone, providing a more accurate geometric foundation for synthetic training.

Decomposition and Composition

To test the impact of specific visual features, the authors used Grounding DINO and Segment Anything (SAM) to extract forearms and objects from real images. These elements were then "composed" onto synthetic hands. This modular approach allowed the team to quantify exactly how much the presence of a forearm or a real-world object improved model performance.

Amplitude Spectrum Augmentation (ASA)

Analysis revealed that synthetic images possess significantly less variance in high-frequency components compared to real photos. To bridge this statistical gap, the authors applied Amplitude Spectrum Augmentation (ASA). This technique distorts the amplitude information of synthetic images while preserving the phase (the underlying structure), forcing the model to become robust against the "unnatural cleanliness" of synthetic data.

VAE Occlusion Prior

To address the challenge of hands interacting with objects, the researchers implemented a Variational Auto-Encoder (VAE) that learns the underlying "rules" of hand poses. This allows the model to effectively "hallucinate" the positions of occluded joints based on visible structural cues.

Experimental Setup and Datasets

The study utilized several key benchmarks for evaluation:

  • Real-world Datasets: FreiHAND (green screen), Dex-YCB (object interaction), HO3D-v2, and MOW (in-the-wild manipulation).
  • Synthetic Datasets: The authors introduced SynFrei and SynDex, synthetic counterparts to major real-world datasets generated via their new pipeline.
  • Baselines: Results were compared against existing synthetic datasets such as DARTset and RenderIH.

Key Insights

The Forearm as a "North Star"

A critical finding of the study is that including a forearm in synthetic data is essential. Without it, models frequently mistake parts of the forearm for the wrist, leading to substantial global rotation errors. The forearm serves as a vital spatial reference for the model.

The 40% Rule and Pose Variety

The authors discovered that a perfect replica of every real-world pose is not required for high performance. Utilizing just 40% of the pose variety found in real datasets allowed synthetic models to reach 97% of real-world performance levels, suggesting that data quality and structural diversity are more important than exhaustive quantity.

Skeleton Topology Alignment

A significant portion of the domain gap is attributed to "label adaptation" rather than visual differences. Different datasets often define joint locations in slightly different physical spots. Aligning these topologies is a prerequisite for any fair evaluation of synthetic-to-real performance.

Results

The authors demonstrated that their synthetic data is nearly indistinguishable from real data in terms of training efficacy:

  • Bridging the Gap: On the FreiHAND benchmark, a model trained on the SynFrei dataset achieved a PA-MPJPE (error) of 1.02 cm, nearly matching the 0.99 cm achieved by training on real data.
  • Outperforming Baselines: The proposed models significantly outperformed those trained on previous synthetic datasets like DARTset (e.g., 0.85 cm error vs 2.56 cm for the CMR method).
  • Generalization: Models trained on a hybrid of real and synthetic data showed superior generalization to "in-the-wild" images (MOW dataset) compared to those trained on real data alone, indicating a beneficial regularization effect from synthetic data.

Conclusion and Future Work

While the study successfully bridges the synthetic-to-real gap, the authors note that current generative models (such as Diffusion-based ControlNet) are not yet a complete solution. While these models produce high-quality images, they often fail to maintain the precise alignment between the hand and the underlying 3D labels required for pose estimation. Future research will focus on improving the alignment of generative AI for data synthesis and automating the adaptation of skeleton topologies across different camera rigs and hand models.

Summary updated

BibTeX
@inproceedings{zhao2025analyzing,
  author = {Zhuoran Zhao and Linlin Yang and Pengzhan Sun and Pan Hui and Angela Yao},
  title = {{Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01144},
}
Figure