Abstract
Synthesizing realistic human-object interaction (HOI) sequences remains a significant challenge due to the disparate spatial scales of body and finger movements and the temporal complexity of maintaining stable grasps on moving objects. We present DiffGrasp, a unified diffusion-based framework that generates whole-body grasping motions conditioned on 3D object shape and motion trajectories. By integrating contact-aware losses and inference-time guidance, DiffGrasp achieves state-of-the-art performance in grasping accuracy and physical realism, significantly outperforming existing two-stage or static methods.
Introduction
Generating realistic human-object interaction is a "holy grail" for animation, VR/AR, and robotics. However, current state-of-the-art models often struggle with two primary issues. First is the Scale Gap: models frequently focus on coarse body movements like walking while ignoring the fine-grained finger movements required for a realistic grasp. Second is the Temporal Gap: models may predict a single static "perfect grasp" but fail to maintain that contact naturally as the object moves through space.
DiffGrasp addresses these challenges by treating the body, hands, and object motion as a unified system within a single diffusion model. This approach avoids the network degradation common in multi-stage pipelines and ensures that fine-grained hand movements are spatially and temporally consistent with global body motion.
Methods
The core of DiffGrasp is a conditional diffusion model designed to perceive and react to the object's geometry and trajectory.
Unified Diffusion Architecture
Unlike previous works that separate hand and body prediction, DiffGrasp employs a single Transformer-based denoiser. The model is conditioned on three primary inputs:
- Object Shape: Represented via Basis Point Sets (BPS) to capture geometric nuances.
- Object Motion: The 3D trajectory and rotation of the object over time.
- Human Identity: SMPL-X parameters defining the character's shape and gender.
Contact-Aware Training and Guidance
To ensure stable hand-object interactions, the authors introduced specialized training losses and inference-time guidance:
- Contact-aware Losses: These include a reconstruction loss that prioritizes hand joint accuracy during expected contact and an interaction loss that uses exponentially decaying distance weights to sensitize the model to the proximity between fingers and the object surface.
- Grasp Stabilization Guidance: During inference, this data-driven term prevents the hand from sliding relative to the object once a grasp is established by "locking" the hand to the object's local coordinate system.
- Hand-Object Contact Guidance: This uses Signed Distance Fields (SDF) to penalize mesh penetration and encourage a snug fit between the hand and the object.
Key Results
DiffGrasp was evaluated on the GRAB and ARCTIC datasets, specifically testing on unseen objects to evaluate generalization. The model demonstrated a massive leap in performance over the baseline (OMOMO):
- Grasping Accuracy: On the GRAB dataset, DiffGrasp achieved an F1 score of 0.7840, compared to the baseline's 0.1028.
- Contact Precision: The mean distance between fingers and the object surface was reduced to just 0.04 cm, a significant improvement over the baseline's 0.15–0.25 cm range.
- Qualitative Realism: As shown in the sequences below, the model successfully performs complex motions, such as stooping to pick up objects, while maintaining stable contact throughout the trajectory.
Discussion and Future Work
A key insight of this work is that joint modeling of the body and hands is essential for preventing the errors that accumulate in decoupled systems. Furthermore, predicting wrist translation relative to the object's centroid makes the grasp significantly more robust to rapid object movements.
Despite these advancements, some limitations remain. The model currently lacks constraints to prevent self-penetration (e.g., an arm passing through the torso) and does not yet support complex locomotion like walking while carrying an object. Future research will focus on integrating physical simulation constraints to resolve penetration issues and expanding the motion vocabulary to include dynamic carrying tasks.
Summary updated