TL;DR
The HomeRobot project introduces a new, standardized benchmark and software stack for Open-Vocabulary Mobile Manipulation (OVMM)—the ability for a robot to find and move arbitrary objects in unseen home environments. By providing both a high-fidelity simulation environment and a reproducible real-world setup using the affordable Hello Robot Stretch, the authors establish a baseline success rate of 20% in the real world, highlighting perception as the primary bottleneck for future research.
What Problem Does This Paper Address?
The "north star" of robotics has long been the creation of useful household assistants. However, most research simplifies the problem by using discrete action spaces, limited object sets, or single-room environments.
The authors tackle Open-Vocabulary Mobile Manipulation (OVMM), which requires a robot to:
- Understand language: "Move the toy animal from the chair to the table."
- Perceive and Navigate: Find a previously unseen object in a multi-room home.
- Manipulate: Grasp the object and place it accurately on a target receptacle.
The core challenge is integration. Even if individual modules for navigation or grasping work well, combining them into a "full-stack" system that generalizes to new objects and layouts remains incredibly difficult. Furthermore, the lack of standardized hardware and software makes it nearly impossible for different labs to reproduce and compare results.
Methods
The authors propose HomeRobot, a library that provides identical APIs for both simulation and real-world robots. They implement a high-level policy called OVMMAgent, a state machine that sequences five key skills: FindObj, Gaze, Pick, FindRec, and Place.
They evaluate two distinct algorithmic families:
- Heuristic (Model-based): This approach uses a 2D semantic mapping module, frontier-based exploration, and the Fast Marching Method for navigation planning. It relies on the DETIC open-vocabulary detector to identify objects and receptacles.
- Reinforcement Learning (RL): The authors trained four separate skills (FindObject, FindReceptacle, GazeAtObject, and PlaceObject) using Decentralized Distributed Proximal Policy Optimization (DDPPO). While trained with ground-truth simulation data, these policies use DETIC predictions at test time to handle the open-vocabulary requirement.
The system architecture offloads heavy computation (perception and planning) to a desktop GPU, communicating with the robot over a dedicated router to minimize latency.
Datasets
The paper introduces a massive, curated dataset for OVMM:
- Simulation Environment: Built on the Habitat Synthetic Scenes Dataset (HSSD), featuring 200+ human-authored 3D home scenes.
- Object Set: A diverse collection of 2,535 unique objects across 129 categories, aggregated from sources like AI2-Thor and Google Scanned Objects. Objects are split into "seen" and "unseen" categories to test generalization.
- Real-World Benchmark: A controlled 3-room apartment environment with standardized furniture and a specific set of seen and unseen objects (e.g., Rubik's cubes, toy drills) to ensure reproducibility across different labs.
Key Insights and Ideas
- Perception is the Bottleneck: There is a massive performance gap between using ground-truth (GT) segmentation and predicted segmentation (DETIC). In simulation, success rates plummeted from ~15% with GT to <1% with DETIC.
- The "Gaze" Skill is Critical: The authors found that simply navigating to an object isn't enough. A dedicated "Gaze" skill—where the robot centers the object in its frame and adjusts its camera tilt—significantly improved grasping success by ensuring better alignment.
- Exploration vs. Efficiency: RL-based navigation was more efficient when the target was visible, but the heuristic planner was superior at long-horizon exploration in complex, multi-room layouts because it maintains an explicit map of explored regions.
- Sim-to-Real Transfer: The authors demonstrated that navigation and placement skills trained in simulation could successfully transfer to the real world, achieving a 20% overall success rate on the Hello Robot Stretch.
Key Results
- Real-World Success: The RL-based baseline achieved a 20% success rate in the real-world apartment, while the heuristic baseline achieved 15%.
- Sub-task Breakdown: In the real world, both baselines were relatively good at finding the initial object (
FindObj~70%), but struggled more with thePick(35-45%) andPlace(15-20%) phases. - Simulation Benchmarking: The paper provides a comprehensive comparison of HomeRobot against existing benchmarks (like ALFRED or RoboTHOR), showing that HomeRobot is unique in offering a multi-purpose, real-world robotics stack with continuous action spaces and demonstrated sim-to-real capabilities.
Limitations and Future Work
- Grasping Simulation: Currently, the simulation uses a "magic snap" to attach objects to the gripper rather than simulating physical contact. The authors acknowledge that training robust real-world grasping in sim requires more sophisticated physics.
- Natural Language: The current benchmark uses templated instructions. Future work aims to incorporate complex, multi-step natural language commands.
- End-to-End Learning: The current modular state-machine approach is prone to cascading errors. The authors suggest that end-to-end baselines might eventually outperform modular ones if the perception-action loop can be tightened.
- Dynamic Obstacles: The current navigation stack does not handle moving objects or people, which is a requirement for truly useful home assistants.
Summary updated