TL;DR
LIVIN is a new benchmark for spatial and embodied intelligence featuring high-fidelity "digital twins" of 30 real-world, lived-in homes. By combining an agentic human-in-the-loop reconstruction workflow with 12,808 interactable objects, it exposes the significant performance gap current AI models face when moving from clean, synthetic environments to the messy, constrained reality of domestic life.
What Problem Does This Paper Address?
Training robots for the home usually relies on two types of data: artist-designed synthetic scenes (which lack real-world "clutter" logic) or raw 3D scans (which are often non-interactable and lack semantic detail). This creates a "sim-to-real" gap where a robot might navigate a pristine virtual living room perfectly but fail in a real home where a bicycle blocks a hallway or a half-open cabinet door creates a snagging hazard.
The authors argue that "lived-in" arrangements—the specific way humans pile belongings, arrange furniture, and leave doors ajar—impose unique spatial constraints that shape both perception and action. LIVIN aims to bridge this gap by providing a benchmark that is both grounded in real-world observations and fully interactable for robotic simulation.
Methods
The researchers developed a human-in-the-loop (HITL) workflow to convert real-world data (floorplans, metric scans, and multi-view images) into simulation-ready digital twins. The pipeline consists of three main stages:
- Instance Recognition: An agent (GPT-6 Astra) identifies and segments objects across views, selecting the best reference images for reconstruction.
- Architectural Reconstruction: The system builds the "shell" of the home, including fine details like niches, beams, and wall openings.
- Object Generation and Placement:
- Articulated objects (cabinets, drawers) are modeled via Blender code generated by GPT-6 Astra to ensure their kinematic structures (hinges, slides) match the real environment.
- Non-articulated objects are generated using Rodin, a 3D generative model, from "cleaned" reference images where occlusions have been removed by GPT-Image-2.
Crucially, humans review and correct the output at each stage to ensure the digital twin remains a faithful replica of the source home.
Datasets
The primary contribution is LIVIN-Scenes, which includes:
- 30 diverse homes containing 319 rooms.
- 12,808 object instances, including 1,559 articulated objects.
- LIVIN-Bench: A suite of four tasks: 3D object detection, 3D scene reconstruction, humanoid navigation, and loco-manipulation.
Compared to existing datasets like 3D-FRONT or HSSD, LIVIN boasts a significantly higher object density (3.70 objects/m²) and a higher percentage of articulated objects (12.17%), reflecting the complexity of actual domestic environments.
Key Insights and Ideas
- The "Clutter" Challenge: Current 3D detection and reconstruction models struggle immensely with the density of real homes. In cluttered scenes, models often miss small objects entirely or fail to recover accurate geometry due to heavy occlusion.
- Articulation Matters: By explicitly modeling 1,559 articulated objects, the authors show that a robot's ability to interact with the environment (e.g., opening a drawer to reach a tool) is just as critical as its ability to walk across a room.
- Spatial Constraints as Features: In LIVIN, household arrangements aren't just background; they are part of the task. A robot might need to move a chair to clear a path or navigate a narrow passage sideways—behaviors rarely required in sparser synthetic datasets.
Key Results
The authors evaluated several state-of-the-art models across the four benchmark tasks:
- 3D Detection & Reconstruction: Even top-tier models like Boxer and Fire3D showed limitations. For instance, in 3D detection, Boxer achieved the highest F1 score (0.528) but still suffered from limited recall in dense scenes.
- Navigation: GPT-6 Astra outperformed specialized Vision-Language Navigation (VLN) models, achieving a 41.48% success rate. However, it still frequently failed due to "goal-completion mismatch"—getting close to the target but failing to stop at the precise interaction region.
- Loco-manipulation: This proved to be the hardest task. GPT-6 Astra achieved a 61.9% success rate on articulated object manipulation (like opening doors) but struggled with "grasping and placement" (24.4%), often making unintended contact with surrounding clutter before establishing a stable grasp.
Limitations and Future Work
While LIVIN provides a leap in realism, the authors acknowledge several areas for growth:
- Scalability: The current HITL workflow, while high-fidelity, requires significant human effort. Automating the verification steps is a key future goal.
- Dynamic Environments: The current digital twins are static snapshots. Future versions could include dynamic elements like moving humans or pets.
- Material Fidelity: While PBR (Physically Based Rendering) materials are used, the physical properties (friction, mass) are currently proposed by agents and verified by humans; more automated ways to ground these in real-world physics would improve simulation accuracy.
By exposing these challenges, LIVIN serves as a rigorous testbed for the next generation of embodied AI systems designed to operate in the messy, beautiful reality of our homes.
Summary updated