TL;DR
UniHSI is a new framework that enables virtual characters to perform complex, long-horizon interactions with their environment using simple natural language commands. By defining interactions as a "Chain of Contacts" (CoC), the system uses a Large Language Model (LLM) to plan sequences of physical contacts which a unified controller then executes in a physics-based simulator.
What Problem Does This Paper Address?
Human-Scene Interaction (HSI) is a cornerstone of embodied AI and virtual reality, yet creating characters that can naturally interact with objects remains a massive challenge. Most existing methods suffer from two main issues:
- Lack of Versatility: Controllers are often "hard-coded" for specific tasks like "sitting" or "lying down." They struggle to generalize to new actions or combine multiple actions into a long sequence.
- Interface Barriers: It is difficult to translate high-level human intent (e.g., "sit down and watch TV") into the precise joint-level movements required by a physics engine.
Previous physics-based methods often required separate policy networks for every single task and relied heavily on labor-intensive, annotated motion data. UniHSI aims to replace these fragmented systems with a single, unified framework that understands language and executes physically plausible movements without needing task-specific annotations.
Methods
The UniHSI framework consists of two primary components: a high-level LLM Planner and a low-level Unified Controller.
1. Chain of Contacts (CoC)
The "secret sauce" is the Chain of Contacts. The authors observed that almost all interactions can be described by which body part touches which object part. They define a "contact unit" as a five-element tuple: {object, object part, humanoid joint, contact type, relative direction}. A complex action is simply a sequence (a chain) of these units.
2. LLM Planner
The framework leverages the world knowledge of LLMs (like GPT-4) to act as a translator. Given a language command and a description of the scene (e.g., "there is a chair with a seat and a back"), the LLM generates a multi-step CoC plan. For example, "sit on the chair" becomes a plan where the pelvis contacts the seat surface from an "up" direction.
3. Unified Controller & TaskParser
The controller is built upon the Adversarial Motion Prior (AMP) framework, which uses a discriminator trained on unlabeled motion capture data to ensure the character moves like a human.
- TaskParser: This is the core of the controller. It fetches the current step from the CoC and translates it into mathematical objectives (rewards) and observations for the character.
- Adaptive Contact Weights: Because some contacts are harder to achieve than others (e.g., touching a small button vs. sitting on a large bed), the system dynamically adjusts reward weights to focus the character's "effort" on the most challenging parts of the task.
Datasets
- ScenePlan (New): The authors introduced this dataset, containing thousands of task plans generated by LLMs across diverse indoor scenarios. It categorizes tasks into "Simple" (1 object, <3 steps), "Medium" (1 object, >3 steps), and "Hard" (multiple objects).
- Motion Datasets: For the adversarial motion prior, they used SAMP (walking, sitting, lying down) and CIRCLE (reaching movements) to provide the character with a library of natural-looking movements.
- Evaluation Scenes: Scenarios were constructed using objects from PartNet and real-world scanned rooms from ScanNet.
Key Insights and Ideas
- Interaction as a Universal Language: By reducing complex behaviors to a sequence of contact pairs, the authors created a "universal language" that bridges the gap between abstract human reasoning (LLMs) and low-level physical control.
- LLMs as Zero-Shot Annotators: One of the most significant findings is that LLMs can generate high-quality interaction plans without any specific training on HSI data. This effectively makes the framework "interaction annotation-free," bypassing the need for expensive manual labeling of motion sequences.
- Decomposition is Key: Instead of trying to learn a single "sit and watch TV" policy, UniHSI breaks it down. The LLM handles the "what" (the plan), and the controller handles the "how" (the physics), leading to much higher success rates for long-horizon tasks.
Key Results
UniHSI demonstrated superior performance across several benchmarks:
- Success Rates: On the PartNet dataset, UniHSI achieved an 85.5% success rate for simple tasks and 40.5% for hard, multi-object tasks.
- Comparison to Baselines: It significantly outperformed vanilla AMP combinations. For the challenging "Lie Down" task, UniHSI achieved an 81.5% success rate, whereas a standard baseline only managed 21.3%.
- Generalization: The model showed a strong ability to generalize to real-world scanned scenes from ScanNet, maintaining a 73.2% success rate on simple tasks despite the "messier" geometry of real-world data.
- Efficiency: The use of Adaptive Contact Weights was shown to be critical; removing them caused success rates to plummet (e.g., from 85.5% down to 21.2% on simple tasks).
Limitations and Future Work
While impressive, the authors note a few areas for improvement:
- Static Environments: Currently, the framework only supports interaction with fixed objects. It cannot yet handle "dynamic" interactions like picking up a chair and moving it.
- Offline Planning: The LLM generates the plan before the character starts moving. A more integrated approach where the LLM can adjust the plan in real-time based on physical feedback would be more robust.
- Spatial Grasp: LLMs still occasionally struggle with precise spatial relationships, sometimes generating plans that are physically awkward or impossible. Refining the prompt engineering or using vision-language models (VLMs) could mitigate this.
Summary updated