Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception

Cheng-Chun Hsu, Zhenyu Jiang, Yuke Zhu

Abstract

Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objects in everyday life, this work explores building articulation models of indoor scenes through a robot's purposeful interactions in these scenes. Prior work on articulation reasoning primarily focuses on siloed objects of limited categories. To extend to room-scale environments, the robot has to efficiently and effectively explore a large-scale 3D space, locate articulated objects, and infer their articulations. We introduce an interactive perception approach to this task. Our approach, named Ditto in the House, discovers possible articulated objects through affordance prediction, interacts with these objects to produce articulated motions, and infers the articulation properties from the visual observations before and after each interaction. It tightly couples affordance prediction and articulation inference to improve both tasks. We demonstrate the effectiveness of our approach in both simulation and real-world scenes.

Introduction

Traditional robotic mapping techniques, such as SLAM, typically generate "static replicas" of environments. While these snapshots are sufficient for navigation, they are inadequate for manipulation tasks. For a robot to interact effectively with a household—for example, retrieving an item from a refrigerator—it requires more than just the object's location; it needs a functional kinematic model. It must understand that a fridge door swings on a hinge and identify the specific axis of that hinge.

Scaling articulation modeling to room-scale environments presents three primary challenges: the vast search space of a 3D room, the presence of partial observations (where internal structures are hidden until opened), and the need for category-agnostic systems that can handle novel or uniquely shaped furniture. Ditto in the House addresses these challenges by enabling robots to autonomously discover and model articulated objects through purposeful interaction.

Methodology

The framework utilizes a three-stage pipeline designed to transition from passive observation to active physical reasoning.

1. Affordance Prediction

The process begins with the robot processing a 3D point cloud of the scene using a PointNet++ architecture. The goal is to predict an "affordance map" that identifies interaction hotspots—specific points on surfaces likely to result in articulated motion if pushed or pulled. This focuses the robot's attention on functionally relevant areas of the room.

2. Interactive Perception

Once a hotspot is identified, the robot interacts with the object and captures egocentric point clouds both before and after the movement. These observations are processed by an Articulation Network (building on the previous Ditto architecture) to segment the mobile part from the static environment. The network then estimates joint parameters, such as the axis of a revolute hinge or the direction of a prismatic slide.

3. Iterative Refinement

A key innovation of this work is the tight coupling between affordance and inference. Initial interactions often result in only partial movement (e.g., a door slightly ajar), providing insufficient data for high-fidelity modeling. The system uses its initial coarse estimate to plan a second, more informative interaction. By selecting a point further from the estimated joint axis to maximize torque, the robot can open the object fully, revealing previously occluded surfaces and significantly improving the accuracy of the final kinematic model.

Experimental Evaluation

The researchers evaluated the system using both simulated and real-world environments:

  • Simulation: 2,500 scenes from the CubiCasa5K floorplan dataset were rendered in the iGibson simulator, covering 80 different object categories.
  • Real-World: Testing was conducted in a household kitchen using LiDAR and camera data from an iPhone 12 Pro to verify the system's robustness outside of controlled simulations.

Key Results and Insights

The experiments demonstrated that interaction is a powerful teacher for perception. Key findings include:

  • Enhanced Discovery: The method discovered approximately 40% more articulated parts than the 3D AffordanceNet baseline, maintaining higher precision across complex scenes.
  • Improved Accuracy: The system achieved a 45% absolute boost in segmentation Intersection over Union (IoU) compared to existing baselines like ScrewNet.
  • The Power of Refinement: The iterative process proved vital for revolute joints. While initial interactions opened only 10% of joints beyond 30 degrees, the refinement step increased this to 55%, drastically reducing errors in joint axis estimation.

Conclusion and Future Work

Ditto in the House shifts the paradigm from passive mapping to active, functional modeling. By physically changing the world, the robot "un-hides" information essential for manipulation. Future research will focus on integrating navigation to eliminate "teleportation" between hotspots, incorporating more complex grasping physics, and accounting for safety constraints and sensor noise in real-world deployments.

Summary updated

Abstract

Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objects in everyday life, this work explores building articulation models of indoor scenes through a robot's purposeful interactions in these scenes. Prior work on articulation reasoning primarily focuses on siloed objects of limited categories. To extend to room-scale environments, the robot has to efficiently and effectively explore a large-scale 3D space, locate articulated objects, and infer their articulations. We introduce an interactive perception approach to this task. Our approach, named Ditto in the House, discovers possible articulated objects through affordance prediction, interacts with these objects to produce articulated motions, and infers the articulation properties from the visual observations before and after each interaction. It tightly couples affordance prediction and articulation inference to improve both tasks. We demonstrate the effectiveness of our approach in both simulation and real-world scenes. Code and additional results are available at https://ut-austin-rpl.github.io/HouseDitto/
BibTeX
@inproceedings{hsu2023ditto,
  author = {Cheng-Chun Hsu and Zhenyu Jiang and Yuke Zhu},
  title = {{Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=GXgdwFWUJX},
}
Figure