TL;DR
Ditto in the House is a framework that enables robots to build functional, kinematic models of entire rooms by "poking" objects to see how they move. By combining affordance prediction (where to touch) with interactive perception (observing the result of a touch), the system can autonomously discover and model articulated objects like cabinets, fridges, and drawers in complex, unstructured environments.
What Problem Does This Paper Address?
Most robotic mapping techniques (like SLAM) create "static replicas"—frozen snapshots of a room. While these are great for navigation, they are useless for manipulation. If a robot needs to get milk from a fridge, it doesn't just need to know where the fridge is; it needs to know that the door swings on a hinge and where that hinge is located.
Existing research into "articulation models" usually focuses on single objects in isolation or relies on pre-defined categories (e.g., "this is a microwave, so it must have a side-hinge"). Scaling this to a whole house is difficult because:
- Search Space: A room is large; the robot must efficiently find "hotspots" worth interacting with.
- Partial Observations: You can't see the inside of a drawer until it's open.
- Category Agnosticism: The robot should be able to handle a weirdly shaped cabinet it has never seen before.
Methods
The authors propose a three-stage pipeline called Ditto in the House:
- Affordance Prediction: Using a PointNet++ architecture, the robot processes a 3D point cloud of the scene to predict an "affordance map." This map identifies "interaction hotspots"—points on objects that are likely to move if pushed or pulled.
- Interactive Perception: The robot interacts with these hotspots and captures "before" and "after" egocentric point clouds. It uses an Articulation Network (building on the authors' previous work, Ditto) to segment the mobile part from the static environment and estimate the joint parameters (e.g., the axis of a revolute hinge or the direction of a prismatic slide).
- Iterative Refinement: This is the "secret sauce." An initial interaction might only nudge a door open by a few degrees (an "ajar" state), which provides poor data. The system uses its first, coarse estimate to calculate a better interaction—specifically, a point further from the estimated joint axis to maximize torque. This second interaction opens the object fully, revealing occluded surfaces and allowing for a much more accurate final model.
Datasets
- CubiCasa5K: A large-scale floorplan dataset. The researchers used 2,500 of these scenes within the iGibson simulator to train and evaluate the model across 80 object categories.
- Real-World Kitchen: To prove the system works outside of simulation, they tested it in a real household kitchen using LiDAR and camera data from an iPhone 12 Pro.
Key Insights and Ideas
- Closing the Loop: The most significant insight is the tight coupling between affordance and inference. The robot doesn't just guess once; it uses its growing understanding of the object's physics to plan a more informative second "poke."
- Category Independence: Unlike many prior models, Ditto doesn't need to know it's looking at a "refrigerator." It simply looks at the geometry and the resulting motion, making it much more robust to novel furniture designs.
- Interaction as a Teacher: The paper reinforces the idea that perception is not a passive act. By physically changing the world, the robot "un-hides" information (like the inside of a cabinet) that is impossible to see from a static viewpoint.
Key Results
- Discovery: The proposed method discovered approximately 40% more articulated parts than the baseline (3D AffordanceNet) with higher precision.
- Accuracy: It achieved a 45% absolute boost in segmentation IoU (Intersection over Union) compared to baselines like ScrewNet.
- Refinement Impact: The iterative process was highly effective for revolute joints (hinges). In the initial stage, only 10% of joints were opened more than 30 degrees. After the refinement step, this jumped to 55%, leading to significantly lower errors in joint axis estimation.
Limitations and Future Work
The authors acknowledge several "shortcuts" taken in this study that provide clear paths for future research:
- Navigation: The robot was "teleported" between locations. In a real deployment, the robot would need to navigate around obstacles to reach the hotspots.
- Simplified Physics: Interactions were modeled as "pseudo-links" (essentially magic tethers) rather than complex grasping or multi-fingered manipulation.
- Safety and Constraints: The current model doesn't account for self-collisions or the physical limits of the robot's arm during the "pulling" motion.
- Perfect Sensing: The study assumes relatively clean depth data and perfect odometry, which are often noisy in real-world robotics.
Summary updated