DoRO: Disambiguation of Referred Object for Embodied Agents

Pradip Pramanick, Chayan Sarkar, Sayan Paul, Ruddra Dev Roychoudhury, Brojeshwar Bhowmick

Abstract

Natural language instructions for embodied agents are often underspecified or ambiguous given the robot's physical environment. While traditional visual grounding focuses on static, single-view scenarios, mobile robots must navigate and aggregate information across multiple viewpoints to resolve these ambiguities. We present DoRO (Disambiguation of Referred Object), a framework that represents instructions as semantic graphs and aggregates multi-view visual data into a 3D occupancy grid. By comparing the semantic requirements of a command against the grounded instances in a scene, DoRO identifies ambiguities and generates targeted clarifying questions. Experimental results in the AI2Thor simulator demonstrate that DoRO significantly outperforms single-view baselines, achieving a 2x improvement in query accuracy.

1. Introduction

When a user instructs a mobile robot to "Bring me the red cup from the dining table," they often assume the environment is perfectly mapped and the instruction is unique. However, real-world environments are frequently ambiguous: there may be multiple red cups, or perhaps no red cup at all. For an embodied agent, this problem is compounded by the fact that objects may be occluded or visible only from specific angles.

Existing visual grounding systems typically operate on static images, which is insufficient for mobile agents that perceive the world through a stream of visual data. To address this, we introduce DoRO (Disambiguation of Referred Object), a system designed to handle "presumptive" instructions by aggregating multi-view observations and performing graph-based reasoning to identify and resolve ambiguities through human-robot dialogue.

2. The DoRO Framework

The DoRO architecture is divided into three primary components: semantic parsing, spatial aggregation, and discriminative reasoning.

2.1 Phrase-to-Graph Network

To translate natural language into a machine-interpretable format, DoRO employs a BERT-based sequence tagger. This network parses the user's instruction into a "semantic graph." In this graph, the target object serves as the root node, with edges connecting it to various attributes (e.g., color, material) and spatial relations (e.g., "on the table," "near the sofa"). This structured representation allows the system to move beyond simple keyword matching and understand the complex dependencies within a command.

2.2 Multi-view Aggregation and 3D Projection

As the robot moves through the environment, it captures a sequence of 2D images. To create a consistent world model, DoRO projects 2D object detections into a 3D world coordinate system. A key innovation here is the use of a soft-mask approach. Instead of treating a 2D bounding box as a uniform block, the system applies a 2D Gaussian function to the detection. This gives higher weight to the center of the object, effectively filtering out background noise and edge artifacts during the 3D projection.

These weighted observations are then discretized into a 2D occupancy grid (Bird’s Eye View). This grid allows the robot to merge object instances across different frames and timestamps, providing a robust count and location for every detected object in the room.

2.3 Graph Discriminator and Query Generation

The "brain" of the system is the Graph Discriminator, which compares the requested semantic graph against the grounded instances in the occupancy grid. The discriminator classifies the state of the instruction into one of four categories:

  • Confirm: A single instance matches all criteria.
  • Mismatch: Objects are found that match the category but fail on specific attributes.
  • Ambiguity: Multiple instances match the description.
  • Missing: No matching objects are found.

If the system detects an Ambiguity or Mismatch, it identifies the "minimal discriminative feature"—the specific attribute that distinguishes the candidates. It then uses a template-based system to generate a clarifying question, such as: "I see a black and a white lamp. Which one are you referring to?"

3. Experimental Evaluation

The system was evaluated using the AI2Thor simulator across kitchen, living room, and bedroom environments. To rigorously test the disambiguation capabilities, the researchers created 68 custom room configurations where multiple identical or similar objects were strategically placed to create ambiguous scenarios.

The dataset included 6,305 text-label pairs for training the semantic parser and 600 end-to-end sequences for testing. Performance was measured using Ambiguity Accuracy (AA), which tracks the system's ability to correctly identify the state of the scene, and Query Accuracy (QA), which measures the relevance of the generated clarifying questions.

4. Results and Analysis

DoRO demonstrated significant improvements over end-to-end BERT-to-BERT baselines. Notably, the system achieved an Ambiguity Accuracy of 0.91 and a Query Accuracy of 0.77. In contrast, single-view baselines achieved a QA of only 0.37, highlighting the necessity of multi-view aggregation for mobile agents.

The system also proved robust to sensor noise. Even with intentional errors introduced to object centroids or the addition of false positives, the multi-view aggregation maintained high F1 scores for object counting (averaging ~0.87–0.90). This suggests that the soft-masking and temporal aggregation effectively "cancel out" the noise inherent in 2D detectors.

5. Conclusion and Future Work

DoRO provides a robust foundation for human-robot interaction in complex environments. By combining semantic graph representations with 3D spatial aggregation, it allows robots to recognize their own uncertainty and communicate it effectively.

Future work will focus on "closing the loop"—enabling the robot to process the user's response to the clarifying query and execute the task. Additionally, the researchers aim to replace the current pseudo-random exploration with "active" exploration strategies, where the robot moves specifically to gain viewpoints that resolve detected ambiguities. Finally, moving beyond template-based language generation to more fluid Natural Language Generation (NLG) will enhance the naturalness of the robot's communication.

Summary updated

BibTeX
@article{pramanick2022doro,
  author = {Pradip Pramanick and Chayan Sarkar and Sayan Paul and Ruddra Dev Roychoudhury and Brojeshwar Bhowmick},
  title = {{DoRO: Disambiguation of Referred Object for Embodied Agents}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2022.3195198},
}
Figure