Unified Human-Scene Interaction via Prompted Chain-of-Contacts

Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, Jiangmiao Pang

TL;DR

UniHSI is a new framework that enables virtual characters to perform complex, long-horizon interactions with their environment using simple natural language commands. By defining interactions as a "Chain of Contacts" (CoC), the system uses a Large Language Model (LLM) to plan sequences of physical contacts which a unified controller then executes in a physics-based simulator.

What Problem Does This Paper Address?

Human-Scene Interaction (HSI) is a cornerstone of embodied AI and virtual reality, yet creating characters that can naturally interact with objects remains a massive challenge. Most existing methods suffer from two main issues:

  1. Lack of Versatility: Controllers are often "hard-coded" for specific tasks like "sitting" or "lying down." They struggle to generalize to new actions or combine multiple actions into a long sequence.
  2. Interface Barriers: It is difficult to translate high-level human intent (e.g., "sit down and watch TV") into the precise joint-level movements required by a physics engine.

Previous physics-based methods often required separate policy networks for every single task and relied heavily on labor-intensive, annotated motion data. UniHSI aims to replace these fragmented systems with a single, unified framework that understands language and executes physically plausible movements without needing task-specific annotations.

Methods

The UniHSI framework consists of two primary components: a high-level LLM Planner and a low-level Unified Controller.

Figure 1: UniHSI facilitates unified and long-horizon control in response to natural language com- mands, offering notable features such as diverse interactions with a singular object, multi-object interactions, and fine-granularity control.

1. Chain of Contacts (CoC)

The "secret sauce" is the Chain of Contacts. The authors observed that almost all interactions can be described by which body part touches which object part. They define a "contact unit" as a five-element tuple: {object, object part, humanoid joint, contact type, relative direction}. A complex action is simply a sequence (a chain) of these units.

2. LLM Planner

The framework leverages the world knowledge of LLMs (like GPT-4) to act as a translator. Given a language command and a description of the scene (e.g., "there is a chair with a seat and a back"), the LLM generates a multi-step CoC plan. For example, "sit on the chair" becomes a plan where the pelvis contacts the seat surface from an "up" direction.

3. Unified Controller & TaskParser

The controller is built upon the Adversarial Motion Prior (AMP) framework, which uses a discriminator trained on unlabeled motion capture data to ensure the character moves like a human.

  • TaskParser: This is the core of the controller. It fetches the current step from the CoC and translates it into mathematical objectives (rewards) and observations for the character.
  • Adaptive Contact Weights: Because some contacts are harder to achieve than others (e.g., touching a small button vs. sitting on a large bed), the system dynamically adjusts reward weights to focus the character's "effort" on the most challenging parts of the task.
Figure 2: Comprehensive Overview of UniHSI. The entire pipeline comprises two principal com- ponents: the LLM Planner and the Unified Controller. The LLM Planner processes language inputs and background scenario information to generate multi-step plans in the form of CoC. Subsequently, the Unified Controller executes CoC step by step, producing interaction movements.

Datasets

  • ScenePlan (New): The authors introduced this dataset, containing thousands of task plans generated by LLMs across diverse indoor scenarios. It categorizes tasks into "Simple" (1 object, <3 steps), "Medium" (1 object, >3 steps), and "Hard" (multiple objects).
  • Motion Datasets: For the adversarial motion prior, they used SAMP (walking, sitting, lying down) and CIRCLE (reaching movements) to provide the character with a library of natural-looking movements.
  • Evaluation Scenes: Scenarios were constructed using objects from PartNet and real-world scanned rooms from ScanNet.

Key Insights and Ideas

  • Interaction as a Universal Language: By reducing complex behaviors to a sequence of contact pairs, the authors created a "universal language" that bridges the gap between abstract human reasoning (LLMs) and low-level physical control.
  • LLMs as Zero-Shot Annotators: One of the most significant findings is that LLMs can generate high-quality interaction plans without any specific training on HSI data. This effectively makes the framework "interaction annotation-free," bypassing the need for expensive manual labeling of motion sequences.
  • Decomposition is Key: Instead of trying to learn a single "sit and watch TV" policy, UniHSI breaks it down. The LLM handles the "what" (the plan), and the controller handles the "how" (the physics), leading to much higher success rates for long-horizon tasks.

Key Results

UniHSI demonstrated superior performance across several benchmarks:

  • Success Rates: On the PartNet dataset, UniHSI achieved an 85.5% success rate for simple tasks and 40.5% for hard, multi-object tasks.
  • Comparison to Baselines: It significantly outperformed vanilla AMP combinations. For the challenging "Lie Down" task, UniHSI achieved an 81.5% success rate, whereas a standard baseline only managed 21.3%.
  • Generalization: The model showed a strong ability to generalize to real-world scanned scenes from ScanNet, maintaining a 73.2% success rate on simple tasks despite the "messier" geometry of real-world data.
  • Efficiency: The use of Adaptive Contact Weights was shown to be critical; removing them caused success rates to plummet (e.g., from 85.5% down to 21.2% on simple tasks).

Limitations and Future Work

While impressive, the authors note a few areas for improvement:

  • Static Environments: Currently, the framework only supports interaction with fixed objects. It cannot yet handle "dynamic" interactions like picking up a chair and moving it.
  • Offline Planning: The LLM generates the plan before the character starts moving. A more integrated approach where the LLM can adjust the plan in real-time based on physical feedback would be more robust.
  • Spatial Grasp: LLMs still occasionally struggle with precise spatial relationships, sometimes generating plans that are physically awkward or impossible. Refining the prompt engineering or using vision-language models (VLMs) could mitigate this.

Summary updated

Abstract

Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausibility, two pivotal factors, versatile interaction control and the development of a user-friendly interface, require further exploration before the practical application of HSI. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands. This framework is built upon the definition of interaction as Chain of Contacts (CoC): steps of human joint-object part pairs, which is inspired by the strong correlation between interaction types and human-object contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution. To facilitate training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability to real scanned scenes. The project page is at https://github.com/OpenRobotLab/UniHSI .
BibTeX
@inproceedings{xiao2024unified,
  author = {Zeqi Xiao and Tai Wang and Jingbo Wang and Jinkun Cao and Wenwei Zhang and Bo Dai and Dahua Lin and Jiangmiao Pang},
  title = {{Unified Human-Scene Interaction via Prompted Chain-of-Contacts}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=1vCnDyQkjg},
}
Figure