Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, Jethro Tan

Abstract

Comprehension of spoken natural language is an essential component for robots to communicate with human effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures including a wide variety of expressions used in spoken language and (2) inherent ambiguity in interpretation of human instructions. In this paper, we propose the first comprehensive system that can handle unconstrained spoken language and is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep-learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and how higher success rates of the object picking task can be achieved through an interactive clarification process.

Introduction

Effective communication between humans and robots in real-world environments requires the ability to process "unconstrained" spoken language. Unlike traditional systems that rely on rigid, pre-defined command sets, human speech is often informal, diverse, and inherently ambiguous. For instance, an instruction like "grab that brown fluffy thing" or "pick up the tissue box" (when multiple boxes are present) poses significant challenges for standard robotic systems.

This research addresses two primary hurdles: the complexity of natural language structures and the inherent ambiguity of human intent. We present a comprehensive system that integrates class-agnostic object detection with a dialogue-based clarification loop, allowing robots to interact with novel objects and resolve uncertainty through natural conversation.

System Architecture and Methods

The proposed system decomposes the pick-and-place task into five integrated subtasks: speech-to-text transcription (via the Web Speech API), candidate object detection, target selection, destination selection, and interactive clarification.

The system architecture integrates vision (SSD/CNN) and speech (LSTM) processing to determine target objects and destinations.

A modified Single Shot Multibox Detector (SSD) is employed for object detection. Crucially, this detector is class-agnostic, trained to distinguish generic "foreground objects" from the background. This enables the robot to detect items it has never encountered during training (zero-shot recognition). For selection, a neural network architecture fuses visual features (processed via CNN) and linguistic features (processed via LSTM) to calculate similarity scores between detected objects and the transcribed instruction.

Resolving Ambiguity through Dialogue

A key innovation of this system is the margin-based clarification method. If the similarity score of the top-ranked object does not sufficiently exceed that of the second-best candidate, the system identifies an ambiguity. Rather than guessing, the robot highlights the potential candidates on a display and asks the operator for more information. This allows the operator to provide follow-up descriptors, such as spatial relationships or specific physical attributes.

An example of the interactive clarification process where the robot resolves ambiguity between two similar teddy bears through dialogue.

Experimental Evaluation

To evaluate the system, we introduced the PFN Picking Instructions for Commodities Dataset (PFN-PIC), featuring 1,180 images and over 91,000 natural language instructions. The dataset reflects cluttered, real-world environments where objects often lack common names, necessitating descriptors based on physical attributes or spatial relationships.

Our experiments on a physical FANUC industrial robot arm and in simulation demonstrated the system's robustness:

  • Simulation: The clarification process reduced errors by 39.2%, improving target selection accuracy from 88.0% to 92.7%.
  • Physical Robot: The system achieved a 75.3% target selection accuracy and a 73.1% end-to-end success rate in complex scenes with over 100 different objects.
  • Generalization: The system maintained a 70.6% target selection accuracy for "unknown" objects not present in the training set, validating the class-agnostic approach.

Conclusion and Future Work

The integration of unconstrained language processing and interactive dialogue significantly enhances robotic utility in human-centric environments. While the software pipeline proved robust, physical limitations such as vacuum gripper failures on porous textures or heavy items remain a challenge. Future research will focus on improving visual robustness under varying lighting conditions and extending the architecture to support multiple languages within a single framework.

Summary updated

Abstract

Comprehension of spoken natural language is an essential component for robots to communicate with human effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures including a wide variety of expressions used in spoken language and (2) inherent ambiguity in interpretation of human instructions. In this paper, we propose the first comprehensive system that can handle unconstrained spoken language and is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep-learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and how higher success rates of the object picking task can be achieved through an interactive clarification process.
BibTeX
@inproceedings{hatori2018interactively,
  author = {Jun Hatori and Yuta Kikuchi and Sosuke Kobayashi and Kuniyuki Takahashi and Yuta Tsuboi and Yuya Unno and Wilson Ko and Jethro Tan},
  title = {{Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=H_aTkWpBw-X},
}
Figure