DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model

Yonghao Zhang, Qiang He, Yanguang Wan, Yinda Zhang, Xiaoming Deng, Cuixia Ma, Hongan Wang

Abstract

Synthesizing realistic human-object interaction (HOI) sequences remains a significant challenge due to the disparate spatial scales of body and finger movements and the temporal complexity of maintaining stable grasps on moving objects. We present DiffGrasp, a unified diffusion-based framework that generates whole-body grasping motions conditioned on 3D object shape and motion trajectories. By integrating contact-aware losses and inference-time guidance, DiffGrasp achieves state-of-the-art performance in grasping accuracy and physical realism, significantly outperforming existing two-stage or static methods.

Introduction

Generating realistic human-object interaction is a "holy grail" for animation, VR/AR, and robotics. However, current state-of-the-art models often struggle with two primary issues. First is the Scale Gap: models frequently focus on coarse body movements like walking while ignoring the fine-grained finger movements required for a realistic grasp. Second is the Temporal Gap: models may predict a single static "perfect grasp" but fail to maintain that contact naturally as the object moves through space.

DiffGrasp addresses these challenges by treating the body, hands, and object motion as a unified system within a single diffusion model. This approach avoids the network degradation common in multi-stage pipelines and ensures that fine-grained hand movements are spatially and temporally consistent with global body motion.

Methods

The core of DiffGrasp is a conditional diffusion model designed to perceive and react to the object's geometry and trajectory.

Unified Diffusion Architecture

Unlike previous works that separate hand and body prediction, DiffGrasp employs a single Transformer-based denoiser. The model is conditioned on three primary inputs:

  • Object Shape: Represented via Basis Point Sets (BPS) to capture geometric nuances.
  • Object Motion: The 3D trajectory and rotation of the object over time.
  • Human Identity: SMPL-X parameters defining the character's shape and gender.
The DiffGrasp architecture utilizes a conditional diffusion model with a Transformer denoiser and inference-time guidance.

Contact-Aware Training and Guidance

To ensure stable hand-object interactions, the authors introduced specialized training losses and inference-time guidance:

  1. Contact-aware Losses: These include a reconstruction loss that prioritizes hand joint accuracy during expected contact and an interaction loss that uses exponentially decaying distance weights to sensitize the model to the proximity between fingers and the object surface.
  2. Grasp Stabilization Guidance: During inference, this data-driven term prevents the hand from sliding relative to the object once a grasp is established by "locking" the hand to the object's local coordinate system.
  3. Hand-Object Contact Guidance: This uses Signed Distance Fields (SDF) to penalize mesh penetration and encourage a snug fit between the hand and the object.

Key Results

DiffGrasp was evaluated on the GRAB and ARCTIC datasets, specifically testing on unseen objects to evaluate generalization. The model demonstrated a massive leap in performance over the baseline (OMOMO):

  • Grasping Accuracy: On the GRAB dataset, DiffGrasp achieved an F1 score of 0.7840, compared to the baseline's 0.1028.
  • Contact Precision: The mean distance between fingers and the object surface was reduced to just 0.04 cm, a significant improvement over the baseline's 0.15–0.25 cm range.
  • Qualitative Realism: As shown in the sequences below, the model successfully performs complex motions, such as stooping to pick up objects, while maintaining stable contact throughout the trajectory.
Qualitative sequences showing DiffGrasp synthesizing whole-body motion for grasping a teapot and a complex tool.

Discussion and Future Work

A key insight of this work is that joint modeling of the body and hands is essential for preventing the errors that accumulate in decoupled systems. Furthermore, predicting wrist translation relative to the object's centroid makes the grasp significantly more robust to rapid object movements.

Despite these advancements, some limitations remain. The model currently lacks constraints to prevent self-penetration (e.g., an arm passing through the torso) and does not yet support complex locomotion like walking while carrying an object. Future research will focus on integrating physical simulation constraints to resolve penetration issues and expanding the motion vocabulary to include dynamic carrying tasks.

Summary updated

Abstract

Generating high-quality whole-body human object interaction motion sequences is becoming increasingly important in various fields such as animation, VR/AR, and robotics. The main challenge of this task lies in determining the level of involvement of each hand given the complex shapes of objects in different sizes and their different motion trajectories, while ensuring strong grasping realism and guaranteeing the coordination of movement in all body parts. Contrasting with existing work, which either generates human interaction motion sequences without detailed hand grasping poses or only models a static grasping pose, we propose a simple yet effective framework that jointly models the relationship between the body, hands, and the given object motion sequences within a single diffusion model. To guide our network in perceiving the object's spatial position and learning more natural grasping poses, we introduce novel contact-aware losses and incorporate a data-driven, carefully designed guidance. Experimental results demonstrate that our approach outperforms the state-of-the-art method and generates plausible whole-body motion sequences.
BibTeX
@inproceedings{zhang2025diffgrasp,
  author = {Yonghao Zhang and Qiang He and Yanguang Wan and Yinda Zhang and Xiaoming Deng and Cuixia Ma and Hongan Wang},
  title = {{DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33120},
}
Figure