Introduction
Generating realistic 3D animations of humans interacting with objects (HOI) from simple text prompts is a significant challenge in computer vision. This task requires bridging the semantic gap between low-dimensional text and high-dimensional motion trajectories while maintaining physical reciprocity—ensuring that humans and objects influence each other's movements realistically without "floating" or "clipping."
Methodology: The HOIAnimator Framework
HOIAnimator addresses these challenges through a dual-diffusion architecture designed to synchronize human and object dynamics.
The framework is built on three core pillars:
- Perceptive Diffusion Models (PDM): Unlike single-model approaches, HOIAnimator employs specialized human-centric and object-centric diffusion models. This separation allows the system to optimize human poses (SMPL-H parameters) and object trajectories (translation/rotation) in parallel.
- Perceptive Message Passing (PMP): This mechanism serves as a communication bridge between the two diffusion models. It utilizes an aggregation module to learn residual values, allowing the model to maintain the core motion intent while subtly aligning the human and object to ensure they move in concert.
- Interaction Contact Field (ICF): Instead of rigid collision detection, the ICF uses a probabilistic distribution to predict likely contact points based on text prompts and object affordances. This learnable spatial probability is highly compatible with the denoising process of diffusion models.
Datasets and Data Processing
The model was trained using the Behave and InterCap datasets. To overcome the lack of detailed textual descriptions in these datasets, the authors standardized motions to 30 FPS and used the SpaCy NLP library to generate complete-sentence annotations, which were then manually filtered for quality.
Key Results
HOIAnimator demonstrates superior performance compared to state-of-the-art baselines like MDM, MLD, and InterGen.
Quantitative Performance
On the BEHAVE dataset, HOIAnimator achieved a Top-3 R-Precision of 0.781, outperforming InterGen (0.734). Crucially, it reduced the Vertex Distance to 0.118, a significant improvement over the baseline of 0.523, indicating much tighter and more accurate physical contact.
Qualitative Evaluation
User studies highlighted that HOIAnimator excels in "Semantic Matching" and "Realism." The animations avoid the common "sliding" effect, instead showcasing believable weight shifts and firm contact points, as seen in the detailed interactions with boxes and backpacks.
Limitations and Future Outlook
While HOIAnimator sets a new benchmark for text-to-HOI animation, it currently struggles with long, multi-action sequences and interactions involving multiple objects. Additionally, the model assumes objects are rigid. Future research aims to incorporate "deformation priors" to handle non-rigid objects like pillows or clothing, further expanding the realism of generated 3D environments.
Summary updated