Natural Language Grounding and Grammar Induction for Robotic Manipulation Commands

Muhannad Al-Omari, Paul Duckworth, Majd Hawasly, David C. Hogg, Anthony G. Cohn

Abstract

We present a cognitively plausible system capable of acquiring knowledge in language and vision from pairs of short video clips and linguistic descriptions. The aim of this work is to teach a robot manipulator how to execute natural language commands by demonstration. This is achieved by first learning a set of visual `concepts' that abstract the visual feature spaces into concepts that have human-level meaning. Second, learning the mapping/grounding between words and the extracted visual concepts. Third, inducing grammar rules via a semantic representation known as Robot Control Language (RCL). We evaluate our approach against state-of-the-art supervised and unsupervised grounding and grammar induction systems, and show that a robot can learn to execute never seen-before commands from pairs of unlabelled linguistic and visual inputs.
BibTeX
@inproceedings{alomari2017natural,
  author = {Muhannad Al-Omari and Paul Duckworth and Majd Hawasly and David C. Hogg and Anthony G. Cohn},
  title = {{Natural Language Grounding and Grammar Induction for Robotic Manipulation Commands}},
  booktitle = {RoboNLP @ ACL},
  year = {2017},
  doi = {10.18653/v1/w17-2805},
}
Figure