Interactive Robotic Grasping with Attribute-Guided Disambiguation

Yang Yang, Xibai Lou, Changhyun Choi

Abstract

Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of the partially observable Markov decision process (Attr-POMDP) for disambiguation. The Attr-POMDP utilizes target and attribute scores as the observation model to calculate the expected return of an attribute-based (e.g., "what is the color of the target, red or green?") or a pointing-based (e.g., "do you mean this one?") question. Our disambiguation module runs in real time on a real robot, and the interactive grasping system achieves a 91.43% selection accuracy in the real-robot experiments, outperforming several baselines by large margins.

Introduction

Natural language is the most intuitive way for humans to command robots, but it is notoriously imprecise. Ambiguity arises from two main sources: visual ambiguity, where a scene contains multiple similar objects (e.g., three different red cans), and linguistic ambiguity, where a user’s command is too vague (e.g., "Give me the snack" in a room full of food).

While a robot could simply guess, a wrong grasp is costly in terms of time and potential damage. Conversely, asking too many questions annoys the user. The challenge lies in creating a system that can identify when it is confused, generate simple, discriminative questions, and decide optimally when it has enough information to stop talking and start acting.

Methodology

The proposed system integrates a multi-stage pipeline into a Franka Emika Panda robot, focusing on a mathematical framework for decision-making under uncertainty.

Object Grounding and Attribute Prediction

The system utilizes MAttNet (a modular attention network) to match the user's query to objects detected in the scene. The researchers extended this framework to predict not just a "matching score," but also generic attributes like color and location for every candidate object. These attributes serve as a robust "middle-ground" for communication; they are easier for robots to detect and simpler for humans to confirm than complex, full-sentence descriptions.

The Attr-POMDP Framework

The core innovation is the Attribute-Guided Partially Observable Markov Decision Process (Attr-POMDP). This models the interaction as follows:

  • State: The "true" target object, which is hidden from the robot.
  • Actions: The robot can choose to AskAttr (ask about color/location), AskPoint (point at an object and ask "this one?"), or Grasp.
  • Reward Function: The robot receives a high reward for a correct grasp, a heavy penalty for an incorrect grasp, and a small negative penalty for every question asked to discourage over-interrogation.

To decide the next move, the robot performs a Belief Tree Search, "imagining" potential future interactions to a depth of three steps and choosing the action that maximizes its expected long-term reward. This allows the robot to plan its next move in real-time (approximately 33ms).

Grasp Execution

Once an object is selected with sufficient confidence, a 3D CNN-based generator calculates a 6-DOF (degree of freedom) grasp pose from point cloud data to execute the physical task.

Experimental Evaluation

Datasets

  • RefCOCO: This standard vision-and-language dataset was used to train the MAttNet grounding module to recognize objects and their attributes.
  • Real-Robot Benchmark: A custom testing set of 70 configurations using 28 different household objects (including "unseen" objects not in the training set) was created to evaluate generalization.

Key Results

The Attr-POMDP system demonstrated significant improvements over baseline methods:

  • Accuracy Boost: On the RefCOCO dataset, the system improved selection accuracy from 27.34% (grounding only) to 92.69% after disambiguation.
  • Real-World Success: In physical robot experiments, the system achieved a 91.43% success rate, significantly outperforming baselines like INGRESS-POMDP (85.71%) and FETCH-POMDP (81.43%).
  • Efficiency: The robot resolved most ambiguities by asking an average of only 2.03 questions per task, successfully balancing the trade-off between accuracy and user interruption.

Conclusion and Future Work

By modeling disambiguation as a POMDP, the robot does not simply ask questions until it reaches a fixed confidence threshold. Instead, it mathematically weighs the cost of a wrong grasp against the cost of the user's time.

Future work aims to expand the robot's attribute vocabulary to include size, shape, and texture. Additionally, the researchers plan to explore "online knowledge updates," allowing the robot to remember user-specific attributes (e.g., "my favorite mug") across different sessions, further reducing the need for repetitive disambiguation.

Summary updated

Abstract

Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of the partially observable Markov decision process (Attr-POMDP) for disambiguation. The Attr-POMDP utilizes target and attribute scores as the observation model to calculate the expected return of an attribute-based (e.g., "what is the color of the target, red or green?") or a pointing-based (e.g., "do you mean this one?") question. Our disambiguation module runs in real time on a real robot, and the interactive grasping system achieves a 91.43\% selection accuracy in the real-robot experiments, outperforming several baselines by large margins.
BibTeX
@inproceedings{yang2022interactive,
  author = {Yang Yang and Xibai Lou and Changhyun Choi},
  title = {{Interactive Robotic Grasping with Attribute-Guided Disambiguation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=LNlKQy99Bg},
}
Figure