A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain-computer interfacing

Daniel Kuhner, Lukas Dominique Josef Fiederer, Johannes Aldinger, Felix Burget, Martin Völker, Robin Tibor Schirrmeister, Chau Do, Joschka Boedecker, Bernhard Nebel, Tonio Ball, Wolfram BurgardRobotics Auton. Syst 2019 · DOI · OpenReview

Abstract

Highlights • BCI-controlled autonomous robotic service assistant. • First online brain–computer-interface using deep learning. • Menu-driven language generation based on referring expressions. • Modular ROS-based mobile robot interaction. • Experimental evaluation with a real robot. Abstract As autonomous service robots become more affordable and thus available for the general public, there is a growing need for user-friendly interfaces to control these systems. Control interfaces typically get more complicated with increasing complexity of robotic tasks and environments. Traditional control modalities such as touch, speech or gesture are not necessarily suited for all users. While some users can make the effort to familiarize themselves with a robotic system, users with motor disabilities may not be capable of controlling such systems even though they need robotic assistance most. In this paper, we present a novel framework that allows these users to interact with a robotic service assistant in a closed-loop fashion, using only thoughts. The system is composed of several interacting components: a brain–computer interface (BCI) that uses non-invasive neuronal signal recording and co-adaptive deep learning, high-level task planning based on referring expressions, navigation and manipulation planning as well as environmental perception. We extensively evaluate the BCI in various tasks, determine the performance of the goal formulation user interface and investigate its intuitiveness in a user study. Furthermore, we demonstrate the applicability and robustness of the system in real-world scenarios, considering fetch-and-carry tasks, close human–robot interactions and in presence of unexpected changes. As our results show, the system is capable of adapting to frequent changes in the environment and reliably accomplishes given tasks within a reasonable amount of time. Combined with high-level task planning based on referring expressions and an autonomous robotic system, interesting new perspectives open up for non-invasive BCI-based human–robot interactions.
BibTeX
@article{kuhner2019service,
  author = {Daniel Kuhner and Lukas Dominique Josef Fiederer and Johannes Aldinger and Felix Burget and Martin Völker and Robin Tibor Schirrmeister and Chau Do and Joschka Boedecker and Bernhard Nebel and Tonio Ball and Wolfram Burgard},
  title = {{A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain-computer interfacing}},
  journal = {Robotics Auton. Syst.},
  year = {2019},
  doi = {10.1016/j.robot.2019.02.015},
}

Loading related content...

Figure