TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions

Ilya A. Petrov, Riccardo Marin, Julian Chibane, Gerard Pons-Moll

TL;DR

This paper introduces TriDi, the first unified probabilistic model for simultaneously generating 3D human poses, object poses, and their interactions. By treating these three modalities as a joint distribution, TriDi can handle seven different conditioning scenarios, surpassing previous one-way models in flexibility and performance.

What Problem Does This Paper Address?

Modeling how humans interact with objects in 3D space is crucial for applications like virtual reality, content creation, and robotics. Existing approaches typically tackle this problem from a single perspective: either predicting human actions given an object, or predicting an object's pose given a human. This "one-way" approach requires specialized models for each scenario, making it inflexible and difficult to scale. The authors aim to bridge this gap by creating a single, unified model that can handle all possible conditional relationships between humans, objects, and their interactions.

Methods

TriDi is built upon the UniDiffuser framework, a type of probabilistic diffusion model. Diffusion models work by progressively adding noise to data and then learning to reverse this process, effectively generating new data. TriDi extends this to three modalities: Human (H), Object (O), and Interaction (I).

Key methodological contributions include:

  • Trilateral Diffusion: TriDi models the joint distribution $P(H, O, I)$ using a three-way diffusion process. This allows it to generate samples for any of the seven possible conditional distributions (e.g., $P(H, I|O)$, $P(O|H, I)$) using a single network.
  • Transformer Architecture: The model uses a transformer architecture with token-level attention to capture fine-grained relationships between the human, object, and interaction modalities.
  • Unified Interaction Representation: A novel approach to represent interactions is introduced by jointly embedding body contact maps (which describe physical contact points) and textual descriptions into a shared latent space. This provides both intuitive control for users and detailed guidance for the model.
  • Classifier-Based Guidance: To enforce the crucial physical constraint of contact, a classifier-based guidance mechanism is employed during the diffusion process. This guides the model to ensure realistic human-object contact.

Datasets

The paper trains and evaluates TriDi on the BEHAVE and GRAB datasets. They also explore scalability by extending training to InterCap and OMOMO.

Key Insights and Ideas

  • Unified Modeling is Superior: Treating human, object, and interaction as a single joint distribution, rather than separate conditional models, leads to better generalization and performance across various tasks.
  • Contact is Key for Realism: Explicitly modeling and enforcing physical contact between humans and objects is essential for generating plausible 3D interactions.
  • Text and Contact Maps Complement Each Other: Combining the interpretability of text descriptions with the precise physical information from contact maps creates a powerful and user-friendly way to control interaction generation.
  • Augmentation Matters: Mirroring interactions to create synthetic data significantly improves generalization, especially for addressing biases like right-handedness in datasets.

Key Results

  • State-of-the-Art Performance: TriDi outperforms specialized one-way baselines on metrics like Coverage (COV), Minimum Matching Distance (MMD), and 1-Nearest Neighbor Accuracy (1-NNA), improving COV by up to 47%.
  • Geometrical Consistency: The model achieves high geometrical consistency for both human and object predictions, demonstrating a strong understanding of spatial relationships and interactions.
  • Versatility: TriDi successfully handles all seven operating modes, including scenarios not addressed by previous work (e.g., $P(H, O, I)$).
  • Qualitative Improvements: Generated samples show better alignment with conditions, less interpenetration, and more diversity compared to baselines.
  • User Preference: A user study indicated that TriDi's outputs are frequently preferred over baseline methods and are competitive with ground truth.

Limitations and Future Work

  • Limitations: The authors acknowledge that TriDi is sensitive to data distribution skewness and may not generalize to objects with significantly novel functionalities (e.g., wheelchairs, bowling balls).
  • Future Work: Promising directions include scaling the model to handle multiple humans and objects interacting, integrating scene conditioning, and exploring more complex social situations. The authors also suggest leveraging advancements in data capture for more sophisticated HOI models.

Summary updated

Abstract

Modeling 3D human-object interaction (HOI) is a problem of great interest for computer vision and a key enabler for virtual and mixed-reality applications. Existing methods work in a one-way direction: some recover plausible human interactions conditioned on a 3D object; others recover the object pose conditioned on a human pose. Instead, we provide the first unified model - TriDi which works in any direction. Concretely, we generate Human, Object, and Interaction modalities simultaneously with a new three-way diffusion process, allowing to model seven distributions with one network. We implement TriDi as a transformer attending to the various modalities' tokens, thereby discovering conditional relations between them. The user can control the interaction either as a text description of HOI or a contact map. We embed these two representations into a shared latent space, combining the practicality of text descriptions with the expressiveness of contact maps. Using a single network, TriDi unifies all the special cases of prior work and extends to new ones, modeling a family of seven distributions. Remarkably, despite using a single model, TriDi generated samples surpass one-way specialized baselines on GRAB and BEHAVE in terms of both qualitative and quantitative metrics, and demonstrating better diversity. We show the applicability of TriDi to scene population, generating objects for human-contact datasets, and generalization to unseen object geometry. The project page is available at: https://virtualhumans.mpi-inf.mpg.de/tridi.
BibTeX
@inproceedings{petrov2025tridi,
  author = {Ilya A. Petrov and Riccardo Marin and Julian Chibane and Gerard Pons-Moll},
  title = {{TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure