Abstract
This paper addresses the challenge of humanoid robot teleoperation in a natural indoor environment via a Brain-Computer Interface (BCI). We leverage deep Convolutional Neural Network (CNN) based image and signal understanding to facilitate both real-time object detection and dry-Electroencephalography (EEG) based human cortical brain bio-signals decoding. We employ recent advances in dry-EEG technology to stream and collect the cortical waveforms from subjects while they fixate on variable Steady State Visual Evoked Potential (SSVEP) stimuli generated directly from the environment the robot is navigating. To these ends, we propose the use of novel variable BCI stimuli by utilising the real-time video streamed via the on-board robot camera as visual input for SSVEP, where the CNN detected natural scene objects are altered and flickered with differing frequencies (10Hz, 12Hz and 15Hz). These stimuli are not akin to traditional stimuli - as both the dimensions of the flicker regions and their on-screen position changes depending on the scene objects detected. On-screen object selection via such a dry-EEG enabled SSVEP methodology, facilitates the on-line decoding of human cortical brain signals, via a specialised secondary CNN, directly into teleoperation robot commands (approach object, move in a specific direction: right, left or back). The resulting classification demonstrates high performance with mean accuracy of 85% for the real-time robot navigation experiment across multiple test subjects.
Introduction
Teleoperating a robot via a Brain-Computer Interface (BCI) is a life-changing prospect for individuals with severe motor disabilities, such as Locked-In Syndrome. However, traditional BCI systems face two major hurdles: hardware friction from "wet" EEG systems requiring conductive gels, and artificial interfaces that use static, abstract icons overlaid on a screen. These abstract interfaces force users to look away from the robot's environment to issue commands, making the experience unintuitive and visually fatiguing. This paper addresses these challenges by creating a "natural" interface where the objects the robot sees—like a chair or a bottle—become the control stimuli themselves.
System Architecture and Methodology
The researchers developed a closed-loop system involving a NAO humanoid robot and a Cognionics Quick-20 dry-EEG headset. The workflow integrates computer vision and signal processing to enable real-time navigation.
The system employs a pre-trained Single Shot MultiBox Detector (SSD) CNN to identify objects in the live video stream from the robot. Instead of static buttons, the system overlays flickering polygon boxes directly onto these detected objects. These boxes serve as Steady State Visual Evoked Potential (SSVEP) stimuli, flickering at frequencies of 10Hz, 12Hz, and 15Hz. Because objects move and change size as the robot navigates, these stimuli are "variable" rather than fixed.
A specialized 1D CNN, termed the SSVEP Convolutional Unit (SCU), processes raw dry-EEG signals to recognize which frequency the user is fixating on, effectively "selecting" the object. Once an object is selected, the robot calculates distance and angle using a photogrammetric approach and automatically walks toward the target.
Experimental Setup and Results
The vision model was pre-trained on the COCO (Common Objects in Context) dataset, while the EEG decoder was trained on a custom a priori offline dataset from three healthy subjects. The system was validated through real-time navigation experiments where subjects guided the robot through an indoor environment.
The system achieved a mean accuracy of 85% for real-time robot navigation across multiple subjects, with an Information Transfer Rate (ITR) of 15.2 bits per minute (bpm). While performance was high across the board, results indicated that the 12Hz frequency was slightly harder to classify than 10Hz or 15Hz, highlighting that frequency selection remains a critical factor for BCI reliability.
Discussion and Conclusion
By turning environmental objects into stimuli, the researchers reduced the cognitive load associated with switching between the environment and controls. The use of a 1D CNN proved robust enough to handle the noise inherent in dry-EEG signals without requiring specialized environments like Faraday cages. This study demonstrates that SSVEP stimuli can remain stable and effective even when they change in size and position dynamically on the screen.
Future work will focus on validating the system with individuals with motor impairments, improving the object detector's performance in cluttered environments, and enhancing the decoder's generalization across different subjects to minimize individual calibration needs.
Summary updated