Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback

Yao Zhang, Zhuchenyang Liu, Yu XiaoarXiv 2026 · arXiv

TL;DR

Inhabitant is a training-free framework that enables Large Language Model (LLM) agents to master complex, long-horizon human-scene interactions by providing them with precise spatial control and metric-based feedback. By turning interaction into a closed-loop control problem, it achieves a 98.7% success rate on challenging household tasks where existing "high-level" agents fail.

What Problem Does This Paper Address?

Generating realistic Human-Scene Interaction (HSI) is shifting from simple, isolated actions (like "sit on a chair") to long-horizon sequences (like "tidy the room by putting toys in a basket and cushions on the sofa").

The authors identify a major "interface gap" in current state-of-the-art agents:

  1. Lack of Spatial Control: Most agents use high-level text prompts (e.g., "pick up the cube"). This is too imprecise for fine-grained physical tasks. If the hand misses by a few centimeters, a text prompt cannot easily specify the exact correction needed.
  2. Absence of Quantitative Feedback: Agents typically observe their failures through rendered images. While a picture might show a missed grasp, it doesn't tell the agent exactly how many centimeters away the hand was. Without metric feedback, the agent cannot systematically "debug" its motion.
Figure 1. One step of a household task: picking up a cube from the counter. (a) Existing agents control the motion through a text prompt or a skill and receive no feedback or only rendered frames; after the hand misses the cube, the agent can only reword the prompt, and the hand misses again. (b) Inhabitant measures how far the hand stopped from the cube (quantitative feedback), fixes where the right hand must be at a given frame (spatial control), and grasps the cube on the next try. The frames in (a) come from our agent restricted to text prompts.

Methods

Inhabitant transforms HSI into a closed-loop spatial control problem. It uses an LLM (like Claude 3.5 Opus) as a high-level planner that interacts with a pretrained motion generator called Kimodo and a physics-enabled simulator.

The framework introduces two critical interfaces:

  • Spatial Control: The agent doesn't just provide text; it writes control specifications. This includes root waypoints (where the body should walk), end-effector targets (precise 3D coordinates for hands or feet), and full-body keyframes.
  • Quantitative Feedback: After a "try" in the simulator, the agent can issue MEASURE calls to get metric data—such as the distance between a palm and a handle or the exact coordinates of a fallen object.

The agent operates in an iterative loop:

  1. TRY: Propose a motion segment with specific spatial constraints.
  2. MEASURE: Query the simulator for quantitative results of that try.
  3. REVISE: If the intent wasn't met (e.g., a missed grasp), use the measurements to adjust the coordinates and try again.
  4. COMMIT: Only add the segment to the final sequence once the measurements confirm success.
Figure 2. Overview of Inhabitant. Left: the agent π controls a pretrained motion generator G by writing a control specification c (spatial control) and interaction events e. The scene simulator E executes the motion, and the agent queries measurements M(q, τ) of the candidate segment τ (quantitative feedback) before it tries again, commits or rolls back (Sec. 3.2). Right: one episode with Claude Opus 5.5, 13 committed segments out of 37 tries. Solid bodies: ends of committed segments, numbered in order; hollow bodies: discarded candidates; insets: three of the interactions; check mark: the success rule is satisfied.

Datasets

The authors introduced Inhabit, a new benchmark designed for long-horizon evaluation:

  • 75 Household Tasks: Ranging from simple one-goal tasks to complex three-goal chains (e.g., "pick up toy, throw it in basket, then sit down").
  • 31 Furniture Layouts: Based on the ReplicaCAD apartment dataset.
  • Diverse Goals: Includes placing, throwing, kicking, opening/closing drawers/fridges, and specific body postures (kneeling, lying).
Overview of the Inhabit benchmark tasks and goal types.

Key Insights and Ideas

  • Interfaces Matter More Than Model Size: One of the most striking findings was that even "weaker" LLMs using Inhabitant's spatial/quantitative interfaces outperformed the strongest models (like Claude Opus) when those models were restricted to standard high-level text interfaces.
  • The "Macro-Micro" Hierarchy: Success in long-horizon tasks is hierarchical. Root waypoints are essential for "macro" success (getting the body to the right location), while end-effector targets provide the "micro" precision needed for the actual interaction (grasping).
  • Feedback as a Debugger: Quantitative feedback allows the agent to treat physical interaction like code debugging. Instead of guessing why a "pick up" failed, the agent sees a "14cm offset" and mathematically corrects its next command.

Key Results

  • Near-Perfect Success: Inhabitant achieved a 98.7% success rate on the Inhabit benchmark using Claude Opus 5.5.
  • Baseline Comparison: Existing agentic baselines (InteractAgent, HumanCLAW, GEMS) struggled significantly, with the best baseline reaching only 26.7% success.
  • Ablation Impact: Removing spatial control caused the agent to fail most tasks (~20% success). Removing quantitative feedback led to much less precise motions, doubling the number of body collisions.
Qualitative comparison showing Inhabitant succeeding where baselines fail due to lack of precise control or feedback.

Limitations and Future Work

  • Simulator Dependency: The "try-before-commit" loop requires a simulator where actions can be undone. This makes it ideal for offline applications like animation or synthetic data generation, but it isn't yet ready for real-time, irreversible physical robotics.
  • Kinematic Drive: The body is currently driven kinematically (without simulating balance or joint torques). While the authors showed that 94% of the motions could be followed by a physics-based controller, full physical integration remains future work.
  • Perception: The agent is currently "given" the state description of the scene. Integrating actual visual perception (e.g., from rendered frames) to replace these text-based state descriptions is a logical next step.

Summary updated

Abstract

Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7\% of the tasks with Claude Opus 5.5, and 93.3\% and 82.7\% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.
BibTeX
@misc{zhang2026inhabitant,
  author = {Yao Zhang and Zhuchenyang Liu and Yu Xiao},
  title = {{Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback}},
  year = {2026},
  eprint = {2611.25005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25005},
}
Figure