Living and Searching in the World: Object-Based State Estimation for Mobile Robots

Lawson L. S. WongAAAI 2014 · OpenReview

Abstract

Mobile-manipulation robots performing service tasks in human-centric indoor environments represent a long-standing goal in autonomous systems. To cook, clean, or assist humans, robots must understand their spatial surroundings beyond simple obstacle avoidance. Unlike structured industrial settings, service robots operate with minimal prior information and must navigate the uncertainty introduced by dynamic agents, such as humans moving objects. This paper proposes a representation framework that prioritizes objects, their semantic attributes, and their geometric realizations. By treating objects as the primary unit of state estimation, the framework allows robots to maintain a task-relevant model of the world that accounts for both physical constraints and perceptual uncertainty.

Introduction

Most contemporary mobile robots navigate using occupancy grids or feature-based Simultaneous Localization and Mapping (SLAM). While effective for avoiding walls and localized navigation, these methods treat the environment as a collection of static points or binary "occupied vs. empty" voxels. They lack a conceptual understanding of objects.

For a service robot to execute high-level commands like "clean the kitchen" or "find the coffee mug," it requires a representation that recognizes objects have types, poses, and physical properties. However, transitioning from raw sensor data to a robust object-based world model is fraught with challenges:

  1. Data Association: Determining if a detection from a new angle corresponds to a previously seen object or a new one.
  2. Partial Views: Sensors like the RGB-D cameras on mobile robots provide sparse, noisy, and incomplete snapshots of a scene.
  3. Negative Information: While detectors identify where an object is, they rarely provide a mechanism to confirm where an object is not, which is crucial for efficient searching.

A Framework for Semantic World Modeling

The proposed framework addresses these challenges by building a "Semantic World Model" through several integrated techniques.

Bayesian Nonparametric Batch-Clustering

Rather than hard-coding object identities, the system treats "objects" as statistical clusters within a joint attribute space (incorporating type, pose, and geometry). By utilizing a Bayesian approach, the robot maintains multiple hypotheses regarding the number of objects in a room and their respective locations. This allows the robot to update its belief as it gathers more evidence, effectively resolving data association ambiguities over time.

Hybrid Filtering and the "On-Demand" Merge

To bridge the gap between low-level geometry and high-level semantics, the framework maintains two parallel filters: a semantic filter for objects and a metric filter for occupancy grids. Tracking every dependency between a semantic label and a geometric voxel is computationally intractable. To solve this, the system employs an "on-demand" merge. For instance, if a robot observes an empty table surface (metric free space), it uses that geometric information to prune semantic hypotheses about objects existing in that specific volume. This hybrid approach ensures the robot remains computationally efficient while benefiting from "negative information."

Temporal and Physical Constraints

Indoor environments are rarely static. To handle objects that move—such as a chair being relocated by a human—the framework explores temporal modeling via Dependent Dirichlet Processes (DDP). This allows the robot to estimate the "decay" of its information, determining how much of its previous world view remains valid after it has been away from a room. Furthermore, the model incorporates physical common sense, such as non-interpenetration (two objects cannot occupy the same space) and stability (objects must be supported by a surface), to further refine the state estimate.

Implementation and Results

The research was evaluated using a Willow Garage PR2 mobile manipulation robot equipped with a Microsoft Kinect for 3-D RGB-D sensing. Object recognition was performed using a black-box system (Bingham Procrustean alignment), allowing the framework to remain modular.

Key results from real-world testing include:

  • Coherent Mapping: The system successfully aggregated sparse detections from multiple viewpoints into a single, coherent object-based map.
  • Efficient State Estimation: By factoring the world model into semantic and metric components, the robot maintained a responsive state estimate without being overwhelmed by the complexity of the environment.
  • Inference from Absence: The robot correctly used "negative" observations to update the probability of an object's existence, allowing for more intelligent search behaviors.

Discussion and Future Work

One of the most significant insights of this work is the idea that a robot should not attempt to track everything. Instead, it should identify "task-relevant variables" and ignore irrelevant data until it becomes necessary. This principle of factoring for scalability is essential for long-term autonomy in complex environments.

However, limitations remain. As the number of objects and the volume of space grow, the state space expands combinatorially. Future work will focus on "execution monitoring," where the robot automatically detects mismatches between its internal model and reality, selectively refining its world view to correct errors. Additionally, handling discrete events—such as a door being closed or a tool being moved—remains a complex challenge for maintaining long-term spatial awareness.

Summary updated

Abstract

Mobile-manipulation robots performing service tasks in human-centric indoor environments has long been a dream for developers of autonomous agents. Tasks such as cooking and cleaning require interaction with the environment, hence robots need to know relevant aspects of their spatial surroundings. However, unlike the structured settings that industrial robots operate in, service robots typically have little prior information about their environment. Even if this information was given, due to the involvement of many other agents (e.g., humans moving objects), uncertainty in the complete state of the world is inevitable over time. Additionally, most information about the world is irrelevant to any particular task at hand. Mobile manipulation robots therefore need to continuously perform the task of state estimation, using perceptual information to maintain the state, and its uncertainty, of task-relevant aspects of the world. Because indoor tasks frequently require the use of objects, objects should be given critical emphasis in spatial representations for service robots. Compared to occupancy grids and feature-based maps often used in navigation and SLAM, object-based representations are arguably still in their infancy. In my thesis, I propose a representation framework based on objects, their 'semantic' attributes, and their geometric realizations in the physical world.
BibTeX
@inproceedings{wong2014living,
  author = {Lawson L. S. Wong},
  title = {{Living and Searching in the World: Object-Based State Estimation for Mobile Robots}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  url = {https://openreview.net/forum?id=S1Z7C6ed-r},
}
Figure