HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation

Wentian Qu, Jiahe Li, Jian Cheng, Jian Shi, Chenyu Meng, Cuixia Ma, Hongan Wang, Xiaoming Deng, Yinda ZhangAAAI 2025 · DOI

Abstract

HOGSA (Hand-Object Gaussian Splatting Augmentation) is a novel framework that utilizes 3D Gaussian Splatting to generate massive, photorealistic synthetic datasets for bimanual hand-object interactions. By integrating mesh-constrained rendering with a physics-aware pose optimization module, HOGSA significantly improves the accuracy and robustness of AI models in tracking complex, two-handed tasks, particularly under heavy occlusion.

Introduction

Understanding how two hands interact with an object is a cornerstone of robotics and virtual reality. However, bimanual tasks involve extreme self-occlusion and a high degree of freedom, making them notoriously difficult to model. The primary bottleneck is the lack of high-quality, large-scale datasets. Collecting and accurately annotating 3D data for two hands is labor-intensive, and existing synthetic methods often fail to achieve the physical realism necessary for effective model generalization.

HOGSA improves pose estimation by providing diverse, photorealistic synthetic data that fills gaps in original training sets.

Methods

The HOGSA framework breaks the data augmentation process into three sophisticated stages designed to ensure both visual fidelity and physical plausibility.

The HOGSA pipeline: from original data to mesh-based Gaussian Splatting, pose optimization, and super-resolution refinement.

1. Mesh-based 3D Gaussian Splatting (3DGS)

Unlike standard point-cloud-based 3DGS, which can produce "floaty" artifacts, HOGSA anchors Gaussian kernels to 3D meshes (using the MANO-HD model for hands). This ensures that rendered textures stay perfectly aligned with the underlying geometry even during complex, high-speed movements.

2. Pose Optimization Module (POM)

To expand the dataset, HOGSA perturbs existing poses and uses an optimization loop to ensure the new configurations are physically plausible. Specific loss functions are employed to prevent hand-object interpenetration and to maintain realistic contact points.

3. Super-Resolution Module (SRM)

To address potential blurriness in 3DGS outputs, a CNN-based refiner (inspired by StyleUNet) is integrated to sharpen the final images. This makes the synthetic data nearly indistinguishable from real captures in terms of texture and detail.

Key Results and Insights

HOGSA was evaluated on two major benchmarks: Arctic (articulated objects) and H2O (rigid objects in first-person views). By applying HOGSA, the researchers expanded these training sets by 113% and 175%, respectively.

  • Pose Accuracy: On the Arctic dataset, the Mean Per-Joint Position Error (MPJPE) dropped from 23.01mm to 20.96mm.
  • Contact Success: The Success Rate (SR) for object manipulation improved from 71.77% to 77.85%.
  • Efficiency: HOGSA generates an augmented image in just 0.06 seconds, significantly faster than diffusion-based competitors (4.5 seconds), enabling the generation of millions of frames.

A key insight is that geometry-anchored splatting prevents texture "drift," while the Pose Optimization Module fills critical gaps in the original dataset's pose distribution, allowing models to "see" through occlusions more effectively.

Conclusion

HOGSA represents a significant step forward in bimanual interaction understanding. While it currently relies on high-quality initial meshes and focuses on hand-object relationships, future work will explore integrating complex dynamic backgrounds and generalizing to entirely unseen object categories.

Summary updated

BibTeX
@inproceedings{qu2025hogsa,
  author = {Wentian Qu and Jiahe Li and Jian Cheng and Jian Shi and Chenyu Meng and Cuixia Ma and Hongan Wang and Xiaoming Deng and Yinda Zhang},
  title = {{HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32712},
}
Figure