Abstract
Modeling 3D human-object interaction (HOI) is a problem of great interest for computer vision and a key enabler for virtual and mixed-reality applications. Existing methods work in a one-way direction: some recover plausible human interactions conditioned on a 3D object; others recover the object pose conditioned on a human pose. Instead, we provide the first unified model—TriDi—which works in any direction. Concretely, we generate Human, Object, and Interaction modalities simultaneously with a new three-way diffusion process, allowing to model seven distributions with one network. We implement TriDi as a transformer attending to the various modalities' tokens, thereby discovering conditional relations between them. The user can control the interaction either as a text description of HOI or a contact map. We embed these two representations into a shared latent space, combining the practicality of text descriptions with the expressiveness of contact maps. Using a single network, TriDi unifies all the special cases of prior work and extends to new ones, modeling a family of seven distributions. Remarkably, despite using a single model, TriDi generated samples surpass one-way specialized baselines on GRAB and BEHAVE in terms of both qualitative and quantitative metrics, and demonstrating better diversity.
Introduction: The Challenge of Unified HOI Modeling
In 3D computer vision, modeling human-object interaction (HOI) is notoriously difficult. Traditional approaches are fragmented, relying on "one-way" specialized models that either predict a human pose given an object or vice-versa. This compartmentalization fails to capture the mutual dependency between the person, the object, and the specific nature of their interaction—such as the difference between sitting on a chair versus lifting it. Furthermore, these methods often struggle with physical plausibility, leading to artifacts like floating objects or mesh interpenetration. TriDi addresses these limitations by introducing a single, unified generative framework capable of modeling the joint distribution of humans, objects, and interactions.
Methods: Trilateral Diffusion
The core innovation of TriDi is Trilateral Diffusion. Building on the UniDiffuser paradigm, the model treats the Human ($H$), the Object ($O$), and the Interaction ($I$) as three distinct but interconnected modalities.
Unified Architecture and Representation
TriDi utilizes a transformer-based architecture that processes tokens representing all three modalities simultaneously. By learning the joint distribution $P(H, O, I)$, the model can handle seven different conditioning configurations at inference time. For instance, it can generate a human for a given object, an object for a given pose, or both from a text description.
A standout feature is the unified interaction representation. TriDi embeds high-level text descriptions (e.g., "thighs are in contact with chair") and low-level body contact maps into a shared latent space. This allows for both intuitive semantic control and precise physical constraints. To further ensure realism, the model employs self-supervised guidance during the denoising process, nudging meshes to align with predicted contact points.
Results and Evaluation
TriDi was trained and evaluated on several key 3D HOI benchmarks, including BEHAVE (real-world scans of large objects) and GRAB (whole-body grasping of smaller objects).
Quantitative and Qualitative Performance
The model demonstrated significant improvements over specialized baselines:
- Distribution Coverage: TriDi improved coverage (COV) by up to 47% on BEHAVE and GRAB, indicating higher diversity in generated samples.
- Human Preference: In user studies, TriDi’s generations were preferred in approximately 89% of cases. Users rated TriDi's outputs as being on par with ground-truth data 52% of the time.
- Generalization: The model successfully generalizes to unseen object geometries, such as figuring out how to sit on a stool after training only on four-legged chairs.
Conclusion and Future Work
TriDi represents a shift from specialized HOI tools to a holistic "world model" of human-object relationships. By treating interaction as a first-class citizen alongside humans and objects, it achieves superior realism and diversity. While currently limited to single human-object pairs and static poses, future work aims to scale the trilateral approach to crowded scenes and continuous 4D motion.
Summary updated