LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

Peijun Xu, Chuansen Nie, Yiyang He, Yinuo Bai, Jingyang Liu, Kuixiang Shao, Yuyang Jiao, Kuanhao Xia, Jiayi Zhu, Zitian Yang, Yanqi Zhang, Tianye Tan, Shuwei Di, Junyi Xu, Jingyi Yu, Jiayuan Gu

TL;DR

LIVIN is a new benchmark for spatial and embodied intelligence featuring high-fidelity "digital twins" of 30 real-world, lived-in homes. By combining an agentic human-in-the-loop reconstruction workflow with 12,808 interactable objects, it exposes the significant performance gap current AI models face when moving from clean, synthetic environments to the messy, constrained reality of domestic life.

What Problem Does This Paper Address?

Training robots for the home usually relies on two types of data: artist-designed synthetic scenes (which lack real-world "clutter" logic) or raw 3D scans (which are often non-interactable and lack semantic detail). This creates a "sim-to-real" gap where a robot might navigate a pristine virtual living room perfectly but fail in a real home where a bicycle blocks a hallway or a half-open cabinet door creates a snagging hazard.

The authors argue that "lived-in" arrangements—the specific way humans pile belongings, arrange furniture, and leave doors ajar—impose unique spatial constraints that shape both perception and action. LIVIN aims to bridge this gap by providing a benchmark that is both grounded in real-world observations and fully interactable for robotic simulation.

Methods

The researchers developed a human-in-the-loop (HITL) workflow to convert real-world data (floorplans, metric scans, and multi-view images) into simulation-ready digital twins. The pipeline consists of three main stages:

  1. Instance Recognition: An agent (GPT-6 Astra) identifies and segments objects across views, selecting the best reference images for reconstruction.
  2. Architectural Reconstruction: The system builds the "shell" of the home, including fine details like niches, beams, and wall openings.
  3. Object Generation and Placement:
    • Articulated objects (cabinets, drawers) are modeled via Blender code generated by GPT-6 Astra to ensure their kinematic structures (hinges, slides) match the real environment.
  4. Non-articulated objects are generated using Rodin, a 3D generative model, from "cleaned" reference images where occlusions have been removed by GPT-Image-2.

Crucially, humans review and correct the output at each stage to ensure the digital twin remains a faithful replica of the source home.

Figure 1: Overview of LIVIN. Built from 30 lived-in homes, LIVIN captures realistic household layouts, furnishings, and daily-use objects, enabling evaluation of 3D detection, reconstruction, nav- igation, and loco-manipulation in realistic domestic environments.

Datasets

The primary contribution is LIVIN-Scenes, which includes:

  • 30 diverse homes containing 319 rooms.
  • 12,808 object instances, including 1,559 articulated objects.
  • LIVIN-Bench: A suite of four tasks: 3D object detection, 3D scene reconstruction, humanoid navigation, and loco-manipulation.

Compared to existing datasets like 3D-FRONT or HSSD, LIVIN boasts a significantly higher object density (3.70 objects/m²) and a higher percentage of articulated objects (12.17%), reflecting the complexity of actual domestic environments.

Key Insights and Ideas

  • The "Clutter" Challenge: Current 3D detection and reconstruction models struggle immensely with the density of real homes. In cluttered scenes, models often miss small objects entirely or fail to recover accurate geometry due to heavy occlusion.
  • Articulation Matters: By explicitly modeling 1,559 articulated objects, the authors show that a robot's ability to interact with the environment (e.g., opening a drawer to reach a tool) is just as critical as its ability to walk across a room.
  • Spatial Constraints as Features: In LIVIN, household arrangements aren't just background; they are part of the task. A robot might need to move a chair to clear a path or navigate a narrow passage sideways—behaviors rarely required in sparser synthetic datasets.
Examples of articulated objects in LIVIN

Key Results

The authors evaluated several state-of-the-art models across the four benchmark tasks:

  • 3D Detection & Reconstruction: Even top-tier models like Boxer and Fire3D showed limitations. For instance, in 3D detection, Boxer achieved the highest F1 score (0.528) but still suffered from limited recall in dense scenes.
  • Navigation: GPT-6 Astra outperformed specialized Vision-Language Navigation (VLN) models, achieving a 41.48% success rate. However, it still frequently failed due to "goal-completion mismatch"—getting close to the target but failing to stop at the precise interaction region.
  • Loco-manipulation: This proved to be the hardest task. GPT-6 Astra achieved a 61.9% success rate on articulated object manipulation (like opening doors) but struggled with "grasping and placement" (24.4%), often making unintended contact with surrounding clutter before establishing a stable grasp.
Qualitative results for 3D detection and reconstruction

Limitations and Future Work

While LIVIN provides a leap in realism, the authors acknowledge several areas for growth:

  • Scalability: The current HITL workflow, while high-fidelity, requires significant human effort. Automating the verification steps is a key future goal.
  • Dynamic Environments: The current digital twins are static snapshots. Future versions could include dynamic elements like moving humans or pets.
  • Material Fidelity: While PBR (Physically Based Rendering) materials are used, the physical properties (friction, mass) are currently proposed by agents and verified by humans; more automated ways to ground these in real-world physics would improve simulation accuracy.

By exposing these challenges, LIVIN serves as a rigorous testbed for the next generation of embodied AI systems designed to operate in the messy, beautiful reality of our homes.

Summary updated

Abstract

Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction. Existing resources often trade off scale, real-world correspondence, and interaction readiness, leaving a gap in faithful, interactive replicas of how real homes are actually arranged. To this end, we introduce LIVIN, a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes. These replicas preserve observed room layouts, furniture configurations, and everyday belongings. To construct them, we design a human-in-the-loop workflow comprising instance recognition, architectural reconstruction, and object generation and placement, with intermediate results reviewed and corrected by humans against the source observations at each stage. We evaluate four tasks in LIVIN: 3D detection, 3D reconstruction, navigation, and loco-manipulation. Our evaluations show that current methods remain challenged by the dense object arrangements, occlusions, limited free space, and constrained interaction regions found in realistic lived-in homes. We hope LIVIN will help advance embodied AI in real-world homes, from spatial understanding to robotic interaction, and ultimately bring embodied intelligence into everyday home environments.
BibTeX
@misc{xu2026livin,
  author = {Peijun Xu and Chuansen Nie and Yiyang He and Yinuo Bai and Jingyang Liu and Kuixiang Shao and Yuyang Jiao and Kuanhao Xia and Jiayi Zhu and Zitian Yang and Yanqi Zhang and Tianye Tan and Shuwei Di and Junyi Xu and Jingyi Yu and Jiayuan Gu},
  title = {{LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes}},
  year = {2026},
  eprint = {2610.12069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.12069},
}
Figure