Diffgrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model

Yonghao Zhang, Qiang He, Yanguang Wan, Yinda Zhang, Xiaoming Deng, Cuixia Ma, Hongan Wang

Abstract

Generating high-quality whole-body human-object interaction motion sequences is becoming increasingly important in various fields such as animation, VR/AR, and robotics. The main challenge of this task lies in determining the level of involvement of each hand given the complex shapes of objects in different sizes and their different motion trajectories, while ensuring strong grasping realism and guaranteeing the coordination of movement in all body parts. Contrasting with existing work, which either generates human interaction motion sequences without detailed hand grasping poses or only models a static grasping pose, we propose a simple yet effective framework that jointly models the relationship between the body, hands, and the given object motion sequences within a single diffusion model. To guide our network in perceiving the object's spatial position and learning more natural grasping poses, we introduce novel contact-aware losses and incorporate a data-driven, carefully designed guidance. Experimental results demonstrate that our approach outperforms the state-of-the-art method and generates plausible whole-body motion sequences.

Introduction

Generating human motion that interacts naturally with objects is a cornerstone of modern digital interaction. However, "whole-body" synthesis is notoriously difficult due to a significant scale mismatch: the large-scale movements of a torso and the millimeter-scale precision of a fingertip operate at different speeds and levels of detail. Existing methods often suffer from the "floating hand" problem—where detailed finger poses are ignored—or temporal inconsistency, where hands slide or penetrate objects during motion. DiffGrasp addresses these issues by ensuring the body stoops or reaches appropriately while fingers maintain a stable, physically plausible grip on a moving object.

Methodology

The core of DiffGrasp is a conditional diffusion model that learns the joint distribution of human-object motion within a unified architecture. Unlike multi-stage pipelines that can accumulate errors, DiffGrasp utilizes a single Transformer-based denoiser.

The DiffGrasp architecture featuring the Conditional Diffusion Model and inference-time Guidance modules.

Unified Architecture and Contact-Awareness

The model processes object shape (via Basis Point Sets), motion trajectories, and human identity. To enhance spatial perception, the authors introduced two novel losses:

  • Contact-aware Reconstruction Loss: Enforces alignment of hand joints and wrists with ground truth specifically during active contact.
  • Contact-aware Interaction Loss: Uses exponentially decaying weights based on object distance to prioritize hand-object spatial relationships as they approach contact.

Inference-Time Guidance

During the sampling phase, the model employs Grasp Stabilization Guidance. This data-driven constraint prevents hand sliding relative to the object once a grasp is established. Additional guidance terms are used to minimize object penetration and ensure stable foot placement.

Experimental Results

DiffGrasp was evaluated on the GRAB and ARCTIC datasets, focusing on whole-body grasping and dexterous bimanual manipulation. The model significantly outperformed the state-of-the-art OMOMO baseline across nearly all metrics.

Qualitative comparison of DiffGrasp against ground truth and baselines, highlighting the realistic "stooping" effect and superior coordination.

Key Performance Metrics

  • Contact Realism: On the GRAB dataset, DiffGrasp achieved an F1 score of 0.7840 for contact accuracy, a massive improvement over the 0.1028 achieved by the best OMOMO variant.
  • Precision: Mean Per-Joint Position Error (MPJPE) was reduced by approximately 30%.
  • Generalization: When tested on unseen objects, DiffGrasp maintained a high F1 score (0.9233), indicating it learns the underlying logic of grasping rather than memorizing specific geometries.

Discussion and Conclusion

A primary insight of this work is that joint modeling of the body and hands prevents the network degradation common in multi-stage pipelines. Qualitatively, DiffGrasp exhibits superior full-body coordination, such as correctly predicting when a character should stoop to reach a low object.

Despite these advances, limitations remain. The model currently lacks explicit constraints to prevent self-penetration (e.g., an arm passing through the torso) and does not yet support locomotion while grasping. Future research will focus on integrating physical laws and collision detection directly into the diffusion process to address these physical inconsistencies.

Summary updated

Abstract

Generating high-quality whole-body human object interaction motion sequences is becoming increasingly important in various fields such as animation, VR/AR, and robotics. The main challenge of this task lies in determining the level of involvement of each hand given the complex shapes of objects in different sizes and their different motion trajectories, while ensuring strong grasping realism and guaranteeing the coordination of movement in all body parts. Contrasting with existing work, which either generates human interaction motion sequences without detailed hand grasping poses or only models a static grasping pose, we propose a simple yet effective framework that jointly models the relationship between the body, hands, and the given object motion sequences within a single diffusion model. To guide our network in perceiving the object's spatial position and learning more natural grasping poses, we introduce novel contact-aware losses and incorporate a data-driven, carefully designed guidance. Experimental results demonstrate that our approach outperforms the state-of-the-art method and generates plausible whole-body motion sequences.
BibTeX
@inproceedings{zhang2025diffgrasp,
  author = {Yonghao Zhang and Qiang He and Yanguang Wan and Yinda Zhang and Xiaoming Deng and Cuixia Ma and Hongan Wang},
  title = {{Diffgrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33120},
}
Figure