HomeRobot: Open-Vocabulary Mobile Manipulation

Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, Chris Paxton

TL;DR

The HomeRobot project introduces a new, standardized benchmark and software stack for Open-Vocabulary Mobile Manipulation (OVMM)—the ability for a robot to find and move arbitrary objects in unseen home environments. By providing both a high-fidelity simulation environment and a reproducible real-world setup using the affordable Hello Robot Stretch, the authors establish a baseline success rate of 20% in the real world, highlighting perception as the primary bottleneck for future research.

What Problem Does This Paper Address?

The "north star" of robotics has long been the creation of useful household assistants. However, most research simplifies the problem by using discrete action spaces, limited object sets, or single-room environments.

The authors tackle Open-Vocabulary Mobile Manipulation (OVMM), which requires a robot to:

  1. Understand language: "Move the toy animal from the chair to the table."
  2. Perceive and Navigate: Find a previously unseen object in a multi-room home.
  3. Manipulate: Grasp the object and place it accurately on a target receptacle.

The core challenge is integration. Even if individual modules for navigation or grasping work well, combining them into a "full-stack" system that generalizes to new objects and layouts remains incredibly difficult. Furthermore, the lack of standardized hardware and software makes it nearly impossible for different labs to reproduce and compare results.

Open-Vocabulary Mobile Manipulation requires agents to search for a previously unseen object and move it to the correct receptacle.

Methods

The authors propose HomeRobot, a library that provides identical APIs for both simulation and real-world robots. They implement a high-level policy called OVMMAgent, a state machine that sequences five key skills: FindObj, Gaze, Pick, FindRec, and Place.

They evaluate two distinct algorithmic families:

  • Heuristic (Model-based): This approach uses a 2D semantic mapping module, frontier-based exploration, and the Fast Marching Method for navigation planning. It relies on the DETIC open-vocabulary detector to identify objects and receptacles.
  • Reinforcement Learning (RL): The authors trained four separate skills (FindObject, FindReceptacle, GazeAtObject, and PlaceObject) using Decentralized Distributed Proximal Policy Optimization (DDPPO). While trained with ground-truth simulation data, these policies use DETIC predictions at test time to handle the open-vocabulary requirement.

The system architecture offloads heavy computation (perception and planning) to a desktop GPU, communicating with the robot over a dedicated router to minimize latency.

Datasets

The paper introduces a massive, curated dataset for OVMM:

  • Simulation Environment: Built on the Habitat Synthetic Scenes Dataset (HSSD), featuring 200+ human-authored 3D home scenes.
  • Object Set: A diverse collection of 2,535 unique objects across 129 categories, aggregated from sources like AI2-Thor and Google Scanned Objects. Objects are split into "seen" and "unseen" categories to test generalization.
  • Real-World Benchmark: A controlled 3-room apartment environment with standardized furniture and a specific set of seen and unseen objects (e.g., Rubik's cubes, toy drills) to ensure reproducibility across different labs.

Key Insights and Ideas

  • Perception is the Bottleneck: There is a massive performance gap between using ground-truth (GT) segmentation and predicted segmentation (DETIC). In simulation, success rates plummeted from ~15% with GT to <1% with DETIC.
  • The "Gaze" Skill is Critical: The authors found that simply navigating to an object isn't enough. A dedicated "Gaze" skill—where the robot centers the object in its frame and adjusts its camera tilt—significantly improved grasping success by ensuring better alignment.
  • Exploration vs. Efficiency: RL-based navigation was more efficient when the target was visible, but the heuristic planner was superior at long-horizon exploration in complex, multi-room layouts because it maintains an explicit map of explored regions.
  • Sim-to-Real Transfer: The authors demonstrated that navigation and placement skills trained in simulation could successfully transfer to the real world, achieving a 20% overall success rate on the Hello Robot Stretch.
RL Gaze skill in action: The agent centers the object in its camera frame to improve grasping success.

Key Results

  • Real-World Success: The RL-based baseline achieved a 20% success rate in the real-world apartment, while the heuristic baseline achieved 15%.
  • Sub-task Breakdown: In the real world, both baselines were relatively good at finding the initial object (FindObj ~70%), but struggled more with the Pick (35-45%) and Place (15-20%) phases.
  • Simulation Benchmarking: The paper provides a comprehensive comparison of HomeRobot against existing benchmarks (like ALFRED or RoboTHOR), showing that HomeRobot is unique in offering a multi-purpose, real-world robotics stack with continuous action spaces and demonstrated sim-to-real capabilities.

Limitations and Future Work

  • Grasping Simulation: Currently, the simulation uses a "magic snap" to attach objects to the gripper rather than simulating physical contact. The authors acknowledge that training robust real-world grasping in sim requires more sophisticated physics.
  • Natural Language: The current benchmark uses templated instructions. Future work aims to incorporate complex, multi-step natural language commands.
  • End-to-End Learning: The current modular state-machine approach is prone to cascading errors. The authors suggest that end-to-end baselines might eventually outperform modular ones if the perception-action loop can be tightened.
  • Dynamic Obstacles: The current navigation stack does not handle moving objects or people, which is a requirement for truly useful home assistants.

Summary updated

Abstract

HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. In addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments to grasp novel objects and place them on target receptacles. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic (model-based) baselines and show evidence of sim-to-real transfer. Our baselines achieve a 20% success rate in the real world; our experiments identify ways future research work improve performance. See videos on our website: https://ovmm.github.io/.
BibTeX
@inproceedings{yenamandra2023homerobot,
  author = {Sriram Yenamandra and Arun Ramachandran and Karmesh Yadav and Austin Wang and Mukul Khanna and Theophile Gervet and Tsung-Yen Yang and Vidhi Jain and Alexander William Clegg and John Turner and Zsolt Kira and Manolis Savva and Angel Chang and Devendra Singh Chaplot and Dhruv Batra and Roozbeh Mottaghi and Yonatan Bisk and Chris Paxton},
  title = {{HomeRobot: Open-Vocabulary Mobile Manipulation}},
  booktitle = {LangRob @ CoRL},
  year = {2023},
  url = {https://openreview.net/forum?id=fAUNGnG6Ed},
}
Figure