HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models

Pei LinAAAI 2025 · DOI

TL;DR

This paper introduces HandDiffuse12.5M, the largest dataset to date for capturing strong, temporally consistent two-hand interactions. They also present HandDiffuse, a diffusion-based generative model that can synthesize realistic two-hand motions, controllable via various inputs like keyframes and trajectories, addressing a significant gap in human motion generation research.

What Problem Does This Paper Address?

Generating realistic and controllable motions for two interacting hands is a challenging problem. Existing datasets are often limited in scale, temporal coherence, or the richness of interactions, making them unsuitable for training robust generative models. Furthermore, capturing the complex dynamics of two hands interacting—including self-occlusion and intricate articulations—is difficult and data-intensive. This scarcity hinders progress in areas like AR/VR, robotics, and virtual avatars, where natural hand interactions are crucial for emotional expression and physical interaction.

Methods

The core of HandDiffuse lies in a diffusion model backbone, adapted for the specific task of two-hand interaction generation. The authors propose a modular approach with two denoisers:

  • Single Hand Denoiser (SHDe): This component focuses on learning the local pose characteristics of individual hands. It takes local joint positions and rotations as input and is trained to reconstruct these local motions.
  • Interacting Hands Denoiser (IHDe): This denoiser builds upon the SHDe's output, learning to optimize the dynamic interaction process between the two hands. It incorporates global information and refines local motions.

To enable controllable generation, HandDiffuse utilizes explicit controllers such as keyframes (for motion in-betweening) and wrist trajectories (for trajectory control). They also introduce a novel Interaction Loss to explicitly penalize artifacts and ensure realistic dynamic interactions, comprising:

  • Contact Potential Loss: This term models the elastic potential energy between finger joints, encouraging realistic contact and penalizing penetration.
  • Shape Loss: This ensures the rationality of hand shapes by comparing bone lengths between predicted and ground truth poses.

The model employs two motion representations: local and global, chosen based on the specific control task to optimize performance.

Datasets

The paper introduces HandDiffuse12.5M, a novel, large-scale dataset specifically designed for two-hand interactions. Key features include:

  • Scale: Approximately 12.5 million images from 250K temporal frames (around 2 hours of data).
  • Content: Captures diverse temporal sequences with strong two-hand interactions, including random poses, daily communication, finger dances, and sign language.
  • Modality: Includes synchronous 50-view videos, 3D key points, and hand poses.
  • Annotation: Manually annotated key points for 240K images to address occlusion.

This dataset is highlighted as being larger in scale and richer in interactions than existing datasets like InterHand2.6M and Re:InterHand.

Key Insights and Ideas

  • Dataset is King: The paper strongly emphasizes that the lack of a large-scale, temporally rich dataset was a primary bottleneck for two-hand interaction generation. HandDiffuse12.5M aims to fill this void.
  • Modularity for Complexity: Breaking down the problem into local hand motion (SHDe) and global interaction dynamics (IHDe) proves effective for handling the complexity of two-hand movements.
  • Explicit Interaction Loss: Directly modeling and penalizing unrealistic interactions (like penetration) through a dedicated loss function is crucial for generating high-quality, physically plausible motions.
  • Controllability is Key: The ability to control generated motions via various inputs (keyframes, trajectories) makes the model practical for real-world applications.

Key Results

  • Dataset Superiority: HandDiffuse12.5M is presented as the largest dataset for interacting hands, with a higher ratio of "contacting" hand poses and a shorter average distance between hand joints compared to existing benchmarks.
  • Generative Performance: HandDiffuse outperforms state-of-the-art methods (MDM, InterGen, BUDDI, InterHandGen) in terms of FID, Diversity, and SDF loss across various tasks, including unconditioned generation, motion in-betweening, and trajectory control.
  • Qualitative Improvements: Visual comparisons show HandDiffuse generating more natural, less penetrative, and more interactive motions compared to baselines, which sometimes exhibit static poses or severe joint penetration.
  • Ablation Studies: The results demonstrate the effectiveness of the proposed modular design (SHDe/IHDe), motion representations, and the Interaction Loss.

Limitations and Future Work

  • Limitations: The authors acknowledge that controllable hand motion generation remains a challenging problem. While their method significantly improves results, artifacts can still occur. The paper also mentions that some downstream applications (like 2D keypoint control for reconstruction) are detailed in the appendix due to space limitations.
  • Future Work: The paper suggests that the HandDiffuse12.5M dataset and the proposed HandDiffuse model will stimulate further research in interacting hands motion generation. The authors also mention their generated motions can be used for data augmentation of other datasets and to enhance existing hand motion capture algorithms.

Summary updated

BibTeX
@inproceedings{lin2025handdiffuse,
  author = {Pei Lin},
  title = {{HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32561},
}
Figure