Abstract
Text2HOI is a novel framework for generating realistic 3D hand-object interaction (HOI) sequences from text prompts and object meshes. By decomposing the task into contact map prediction and diffusion-based motion generation, the system produces physically plausible interactions that generalize to unseen object categories.
Introduction
While recent advancements in generative AI have enabled the creation of human body motions from text, generating intricate hand-object interactions remains a significant challenge. Standard models often struggle with the high-precision requirements of HOI, leading to artifacts like hands floating near objects or clipping through surfaces. For example, a prompt like "open a box" requires precise coordination between fingers and the articulated lid.
The primary hurdles in this domain are the scarcity of labeled 3D HOI datasets and the difficulty of maintaining physical plausibility over temporal sequences. Text2HOI addresses these by leveraging geometric priors and a multi-stage generation pipeline.
Methods
The Text2HOI framework consists of three main stages:
- Contact Map Prediction: A VAE-based network analyzes the object's local geometry and the text prompt to estimate contact probabilities. This geometric focus allows the model to handle novel objects by understanding surface properties rather than semantic categories.
- Text-to-3D Motion Generation: A Transformer-based diffusion model generates 3D poses for both hands and the object. It incorporates agent-wise positional encoding to distinguish between the left hand, right hand, and the object across time steps.
- Hand Refinement Module: A post-processing "refiner" adjusts hand joints to minimize penetration and ensure the hands remain close to the object surface, enhancing physical realism.
Datasets
To facilitate training, the authors augmented three existing datasets with descriptive text labels:
- H2O: Focused on bimanual manipulation.
- GRAB: A dataset of whole-body grasping.
- ARCTIC: Involving articulated objects like laptops or drawers.
Key Insights
- Geometric Generalization: By conditioning on local surface geometry, the model learns universal grasping patterns that apply to unseen objects.
- Contact as a Prior: Predicting contact maps before motion generation significantly constrains the search space, leading to more accurate results.
- Agent-Aware Transformers: Explicitly encoding the identity of the interacting agents (hands and object) improves the model's ability to capture complex dependencies.
Results
Text2HOI demonstrates superior performance compared to baselines like MDM and IMOS. On the GRAB dataset, it achieved a physical realism score of 0.8839 and an action classification accuracy of 92.18%.
The model successfully generates plausible motions for unseen objects, such as teddy bears or round bottles, by relying on its geometric understanding.
Conclusion
Text2HOI represents a significant step forward in text-guided 3D motion generation for hand-object interaction. While the current model does not incorporate force modeling (e.g., friction or torque), it provides a robust foundation for future work in physics-based simulation and robotic manipulation.
Summary updated