HOIAnimator: Generating Text-Prompt Human-Object Animations Using Novel Perceptive Diffusion Models

Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hau, Chenglizhao Chen, Ning Li, Hong Qin

Introduction

Generating realistic 3D animations of humans interacting with objects (HOI) from simple text prompts is a significant challenge in computer vision. This task requires bridging the semantic gap between low-dimensional text and high-dimensional motion trajectories while maintaining physical reciprocity—ensuring that humans and objects influence each other's movements realistically without "floating" or "clipping."

Methodology: The HOIAnimator Framework

HOIAnimator addresses these challenges through a dual-diffusion architecture designed to synchronize human and object dynamics.

HOIAnimator generates diverse animations from text prompts, including lifting boxes, resting against tables, and carrying backpacks.

The framework is built on three core pillars:

  1. Perceptive Diffusion Models (PDM): Unlike single-model approaches, HOIAnimator employs specialized human-centric and object-centric diffusion models. This separation allows the system to optimize human poses (SMPL-H parameters) and object trajectories (translation/rotation) in parallel.
  2. Perceptive Message Passing (PMP): This mechanism serves as a communication bridge between the two diffusion models. It utilizes an aggregation module to learn residual values, allowing the model to maintain the core motion intent while subtly aligning the human and object to ensure they move in concert.
  3. Interaction Contact Field (ICF): Instead of rigid collision detection, the ICF uses a probabilistic distribution to predict likely contact points based on text prompts and object affordances. This learnable spatial probability is highly compatible with the denoising process of diffusion models.

Datasets and Data Processing

The model was trained using the Behave and InterCap datasets. To overcome the lack of detailed textual descriptions in these datasets, the authors standardized motions to 30 FPS and used the SpaCy NLP library to generate complete-sentence annotations, which were then manually filtered for quality.

Key Results

HOIAnimator demonstrates superior performance compared to state-of-the-art baselines like MDM, MLD, and InterGen.

Detail of a generated animation showing realistic hand placement and weight distribution when lifting a large box.

Quantitative Performance

On the BEHAVE dataset, HOIAnimator achieved a Top-3 R-Precision of 0.781, outperforming InterGen (0.734). Crucially, it reduced the Vertex Distance to 0.118, a significant improvement over the baseline of 0.523, indicating much tighter and more accurate physical contact.

Qualitative Evaluation

User studies highlighted that HOIAnimator excels in "Semantic Matching" and "Realism." The animations avoid the common "sliding" effect, instead showcasing believable weight shifts and firm contact points, as seen in the detailed interactions with boxes and backpacks.

Limitations and Future Outlook

While HOIAnimator sets a new benchmark for text-to-HOI animation, it currently struggles with long, multi-action sequences and interactions involving multiple objects. Additionally, the model assumes objects are rigid. Future research aims to incorporate "deformation priors" to handle non-rigid objects like pillows or clothing, further expanding the realism of generated 3D environments.

Summary updated

Abstract

To date, the quest to rapidly and effectively produce human-object interaction (HOI) animations directly from textual descriptions stands at the forefront of computer vision research. The underlying challenge demands both a discriminating interpretation of language and a comprehen-sive physics-centric model supporting real-world dynamics. To ameliorate, this paper advocates HOIAnimator, a novel and interactive diffusion model with perception ability and also ingeniously crafted to revolutionize the animation of complex interactions from linguistic narratives. The effectiveness of our model is anchored in two ground-breaking innovations: (1) Our Perceptive Diffusion Models (PDM) brings together two types of models: one focused on hu-man movements and the other on objects. This combination allows for animations where humans and objects move in concert with each other, making the overall motion more realistic. Additionally, we propose a Perceptive Message Passing (PMP) mechanism to enhance the communication bridging the two models, ensuring that the animations are smooth and unified; (2) We devise an Interaction Contact Field (ICF), a sophisticated model that implicitly captures the essence of HOls. Beyond mere predictive contact points, the ICF assesses the proximity of human and object to their respective environment, informed by a probabilistic distribution of interactions learned throughout the denoising phase. Our comprehensive evaluation showcases HOlani-mator's superior ability to produce dynamic, context-aware animations that surpass existing benchmarks in text-driven animation synthesis.
BibTeX
@inproceedings{song2024hoianimator,
  author = {Wenfeng Song and Xinyu Zhang and Shuai Li and Yang Gao and Aimin Hao and Xia Hau and Chenglizhao Chen and Ning Li and Hong Qin},
  title = {{HOIAnimator: Generating Text-Prompt Human-Object Animations Using Novel Perceptive Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00083},
}
Figure