OVAMOS: A Framework for Open-Vocabulary Multi-Object Search in Unknown Environments

Abstract

Multi-Object Search (MOS) in unknown environments is a fundamental robotics challenge characterized by observation instability, occlusions, and high-dimensional search spaces. We introduce OVAMOS, a framework that combines Vision-Language Models (VLMs) with probabilistic planning to enhance search robustness and efficiency. By employing a multi-layer value map with Bayesian-inspired decay and a POUCT-based planner, OVAMOS effectively manages uncertainty and explores unknown frontiers. Evaluated on the Habitat Matterport3D (HM3D) dataset and in real-world office settings, OVAMOS achieves a 55.0% success rate, nearly doubling the performance of previous state-of-the-art methods.

Introduction

Finding multiple items in an unfamiliar, cluttered environment—such as a robot gathering tools in a workshop—requires simultaneous exploration and target localization. Existing Multi-Object Search (MOS) approaches often struggle with three primary issues: observation instability (detectors failing due to viewpoint or occlusion), the lack of prior maps, and the computational "explosion" of Partially Observable Markov Decision Processes (POMDPs) in large-scale environments. Many current systems rely on greedy strategies that fail to revisit high-probability areas after an initial unsuccessful scan.

Methodology

The OVAMOS framework addresses these challenges by bridging high-level semantic reasoning with low-level path planning.

Expand

Source, p. 3

VLM-Guided Value Maps and Bayesian Decay

OVAMOS utilizes BLIP-2 to generate semantic heatmaps based on room appearance, predicting object locations (e.g., a "cup" near a "table"). A key innovation is the Bayesian-inspired value decay. When the robot inspects a high-value region without finding the target, the framework applies a decay function to that region's probability. This prevents the robot from becoming trapped in "semantic loops" and encourages it to seek alternative locations or explore new frontiers.

POUCT Planning

Rather than simply moving to the highest-value pixel, OVAMOS employs Partially Observable Upper Confidence Trees (POUCT). This algorithm simulates future actions and observations within a search tree, balancing the cost of navigation against the potential reward of discovery. The action space unifies "candidate points" (likely object locations) and "frontiers" (edges of the unknown map), allowing for seamless transitions between searching and mapping.

Perception Stack

For object detection, the system integrates an open-vocabulary stack consisting of Grounding DINO, YOLOv7, and the Segment Anything Model (SAM), enabling the robot to identify a wide variety of objects without task-specific retraining.

Experiments and Results

The framework was validated through extensive simulations and physical deployments.

Simulation Results

Using 120 episodes across five scenes from the Habitat Matterport3D (HM3D) dataset, OVAMOS was compared against the state-of-the-art "Finder" method. OVAMOS achieved a 55.0% Success Rate (SR), a significant improvement over Finder's 28.3%. Furthermore, the Multi-Object Success weighted by Path Length (MSPL) increased from 0.198 to 0.497, indicating much higher search efficiency.

Real-World Deployment

A TurtleBot robot executing a multi-object search in a cluttered office environment, navigating between candidate points and frontiers. Source, p. 1

Expand

In a 50-square-meter cluttered office, a TurtleBot equipped with OVAMOS successfully located various items, including a chair, a fridge, and an umbrella. Notably, the umbrella was initially occluded by a trash can; however, the POMDP planner recognized the area remained high-value and directed the robot to revisit it from a different angle, eventually leading to a successful detection.

Conclusion

OVAMOS demonstrates that combining semantic priors from VLMs with robust probabilistic planning significantly improves multi-object search in the wild. Future work will focus on extending the framework to 3D mapping, incorporating active camera control for better scanning, and adding manipulation capabilities to allow robots to move obstacles during their search.

@misc{wang2025ovamos,
  author = {Qianwei Wang and Yifan Xu and Vineet R. Kamat and Carol C. Menassa},
  title = {{OVAMOS: A Framework for Open-Vocabulary Multi-Object Search in Unknown Environments}},
  year = {2025},
  eprint = {2503.02106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02106},
}