Robot Programming by Demonstration with situated spatial language understanding

Maxwell Forbes, Rajesh P. N. Rao, Luke Zettlemoyer, Maya CakmakICRA 2015 · OpenReview

Abstract

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions-our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method's ability to deal with incomplete language.
BibTeX
@inproceedings{forbes2015robot,
  author = {Maxwell Forbes and Rajesh P. N. Rao and Luke Zettlemoyer and Maya Cakmak},
  title = {{Robot Programming by Demonstration with situated spatial language understanding}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2015},
  url = {https://openreview.net/forum?id=MXD6OxDvjt},
}

Loading related content...

Figure