TL;DR
Inhabitant is a training-free framework that enables Large Language Model (LLM) agents to master complex, long-horizon human-scene interactions by providing them with precise spatial control and metric-based feedback. By turning interaction into a closed-loop control problem, it achieves a 98.7% success rate on challenging household tasks where existing "high-level" agents fail.
What Problem Does This Paper Address?
Generating realistic Human-Scene Interaction (HSI) is shifting from simple, isolated actions (like "sit on a chair") to long-horizon sequences (like "tidy the room by putting toys in a basket and cushions on the sofa").
The authors identify a major "interface gap" in current state-of-the-art agents:
- Lack of Spatial Control: Most agents use high-level text prompts (e.g., "pick up the cube"). This is too imprecise for fine-grained physical tasks. If the hand misses by a few centimeters, a text prompt cannot easily specify the exact correction needed.
- Absence of Quantitative Feedback: Agents typically observe their failures through rendered images. While a picture might show a missed grasp, it doesn't tell the agent exactly how many centimeters away the hand was. Without metric feedback, the agent cannot systematically "debug" its motion.
Methods
Inhabitant transforms HSI into a closed-loop spatial control problem. It uses an LLM (like Claude 3.5 Opus) as a high-level planner that interacts with a pretrained motion generator called Kimodo and a physics-enabled simulator.
The framework introduces two critical interfaces:
- Spatial Control: The agent doesn't just provide text; it writes control specifications. This includes root waypoints (where the body should walk), end-effector targets (precise 3D coordinates for hands or feet), and full-body keyframes.
- Quantitative Feedback: After a "try" in the simulator, the agent can issue
MEASUREcalls to get metric data—such as the distance between a palm and a handle or the exact coordinates of a fallen object.
The agent operates in an iterative loop:
- TRY: Propose a motion segment with specific spatial constraints.
- MEASURE: Query the simulator for quantitative results of that try.
- REVISE: If the intent wasn't met (e.g., a missed grasp), use the measurements to adjust the coordinates and try again.
- COMMIT: Only add the segment to the final sequence once the measurements confirm success.
Datasets
The authors introduced Inhabit, a new benchmark designed for long-horizon evaluation:
- 75 Household Tasks: Ranging from simple one-goal tasks to complex three-goal chains (e.g., "pick up toy, throw it in basket, then sit down").
- 31 Furniture Layouts: Based on the ReplicaCAD apartment dataset.
- Diverse Goals: Includes placing, throwing, kicking, opening/closing drawers/fridges, and specific body postures (kneeling, lying).
Key Insights and Ideas
- Interfaces Matter More Than Model Size: One of the most striking findings was that even "weaker" LLMs using Inhabitant's spatial/quantitative interfaces outperformed the strongest models (like Claude Opus) when those models were restricted to standard high-level text interfaces.
- The "Macro-Micro" Hierarchy: Success in long-horizon tasks is hierarchical. Root waypoints are essential for "macro" success (getting the body to the right location), while end-effector targets provide the "micro" precision needed for the actual interaction (grasping).
- Feedback as a Debugger: Quantitative feedback allows the agent to treat physical interaction like code debugging. Instead of guessing why a "pick up" failed, the agent sees a "14cm offset" and mathematically corrects its next command.
Key Results
- Near-Perfect Success: Inhabitant achieved a 98.7% success rate on the Inhabit benchmark using Claude Opus 5.5.
- Baseline Comparison: Existing agentic baselines (InteractAgent, HumanCLAW, GEMS) struggled significantly, with the best baseline reaching only 26.7% success.
- Ablation Impact: Removing spatial control caused the agent to fail most tasks (~20% success). Removing quantitative feedback led to much less precise motions, doubling the number of body collisions.
Limitations and Future Work
- Simulator Dependency: The "try-before-commit" loop requires a simulator where actions can be undone. This makes it ideal for offline applications like animation or synthetic data generation, but it isn't yet ready for real-time, irreversible physical robotics.
- Kinematic Drive: The body is currently driven kinematically (without simulating balance or joint torques). While the authors showed that 94% of the motions could be followed by a physics-based controller, full physical integration remains future work.
- Perception: The agent is currently "given" the state description of the scene. Integrating actual visual perception (e.g., from rendered frames) to replace these text-based state descriptions is a logical next step.
Summary updated