TL;DR
COMBS (Concealed Object Multi-modal Bayesian Search) is a new framework that allows robots to find hidden objects by fusing "common sense" from LLMs, vision-language models, and semantic networks into a principled Bayesian decision model. By treating the environment as a dynamic hierarchy and explicitly reasoning about whether containers are hiding their contents, COMBS achieves near-perfect success rates even when objects are placed in highly unusual locations.
What Problem Does This Paper Address?
Imagine asking a robot to find a fork. It knows to check the kitchen drawer. But what if the fork is hidden inside a box in the bathroom?
Most current robotic search methods struggle with two things: occlusion (objects hidden inside containers) and out-of-distribution (OOD) placements (objects in weird places). While Large Language Models (LLMs) are great at providing "common sense" priors (e.g., "forks are in kitchens"), they can be brittle, slow, and prone to "hallucinating" that an object is absent if it isn't in a typical spot.
The authors address the challenge of building a principled belief system that allows a robot to navigate a partially known environment, decide when to open a container versus when to move to another room, and ultimately determine with statistical confidence if an object is truly missing from the house.
Methods
The COMBS framework rests on three main pillars:
- Hierarchical Transition System (HTS): Instead of a flat map, the robot views the world as a tree. Nodes represent rooms, which contain objects, which might be containers holding more objects. Crucially, this tree grows as the robot interacts with the world—opening a drawer reveals new nodes to search.
- CA-MOMDP (Concealment-Adaptive Mixed-Observability MDP): This is the "brain" of the system. It’s a mathematical framework that tracks two types of hidden information: where the target is, and whether a specific object (like a closed box) is currently concealing its contents.
- Multi-modal Prior Fusion: To avoid over-relying on a single potentially flawed model, COMBS fuses three sources into a "COMBS-score":
- CLIP (VLM): Provides visual-semantic similarity.
- ConceptNet: A human-curated semantic network for "common sense."
- Llama 3 (LLM): Captures linguistic patterns of where things are kept.
These are combined using log-odds pooling, ensuring that no single model can completely zero out a location's probability, which allows the robot to recover if its initial "common sense" guess was wrong.
Datasets
The authors evaluated COMBS across five different simulator configurations to test everything from perfect perception to messy, real-world-style sensor noise:
- Gridworld: Abstract testing of the core logic.
- ProcTHOR: Procedurally generated 3D houses with interactive objects (drawers/cabinets).
- Matterport3D (MP3D): Real-world scans of homes.
- MP3D + Detic: Using a real open-vocabulary detector (Detic) to simulate the noisy, imperfect labels a real robot would see.
Key Insights and Ideas
- The Power of Negative Evidence: One of the most critical features of COMBS is how it handles "not finding" an object. If a robot looks at a closed drawer and doesn't see the fork, COMBS doesn't assume the fork isn't there; it increases the probability that the drawer is concealing the fork. This prevents the robot from prematurely giving up on a room.
- Bounded Influence: By pooling priors in log-odds space, the system ensures robustness. If the LLM is 100% sure the fork is in the kitchen but it's actually in the bathroom, the other models (and subsequent negative observations) eventually override the LLM's bias.
- Principled Termination: Unlike many LLM-based agents that might "quit" randomly, COMBS uses a likelihood ratio to declare an object absent only when the probability of it being anywhere else falls below a specific threshold.
Key Results
COMBS demonstrated significant improvements over baselines like TidyBot and pure LLM policies:
- Success Rate: Found 92–100% of targets across all simulators, including those in "out-of-distribution" locations (like a fork in a bathroom).
- Efficiency: It matched or beat the search speed of baselines while being significantly cheaper to run than per-step LLM queries.
- Robustness to Noise: Even with the noisy Detic detector, COMBS maintained a 94-98% success rate.
- Latency: The decision time for COMBS is in the range of 1.7–3.1 ms, orders of magnitude faster than calling an external LLM at every step (which takes ~500ms).
Limitations and Future Work
The authors note that currently, the HTS treats all instances of a class in a room (e.g., three identical cabinets) as sharing the same belief. This can "dilute" the prior if one cabinet is much more likely to hold the target than the others based on spatial context (e.g., the cabinet next to the sink vs. the one by the door). Future work aims to incorporate spatial reasoning to distinguish between these individual instances. Additionally, while the system handles "absent" targets well, it assumes the target is stationary during the search.
Summary updated