Acquiring neural signals for developing a perception and cognition model

Wei Li, Yunyi Li, Genshe Chen, Dan Shen, Erik Blasch, Khanh Pham, Robert S. LynchProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2012 · DOI

TL;DR

This paper presents a Brain-Computer Interface (BCI) system that uses EEG signals to control a humanoid robot's walking behaviors. The goal is to develop a perception and cognition model by understanding the relationship between human mental activities (imagining walking actions) and the robot's corresponding movements.

What Problem Does This Paper Address?

The paper addresses the challenge of integrating human intelligence into automated systems, particularly for controlling complex robots like humanoids. While BCIs have shown promise for controlling devices, there's a need to understand how human thought processes translate into robot actions, especially for tasks requiring nuanced perception and cognition. This is crucial for applications like personal assistance for individuals with motor disabilities, where seamless human-robot interaction is paramount. The authors aim to bridge the gap between raw neural signals and complex robotic behaviors by developing a model that captures this relationship.

Methods

The core of the system is a BCI that integrates an electroencephalograph (EEG) for brainwave acquisition, a Kumotek KT-X humanoid robot with 20 degrees of freedom (DOFs), and a Cannon VC-C50i CCD camera.

  • EEG Data Acquisition: A 32-channel EEG cap is used to record brain activity from the scalp.
  • Humanoid Robot Control: The KT-X robot is capable of walking, arm movements, and head motion. For this study, three specific walking behaviors were implemented: "turning right," "turning left," and "walking forward."
  • Mental Activity Elicitation: Subjects were instructed to imagine performing these walking actions while simultaneously moving their hands (right hand for turning right, left hand for turning left, both hands for walking forward). This hand movement was used as a physical correlate to help identify the mental activity.
  • Video Analysis: The CCD camera captures video of the subjects' hand postures to precisely identify the time intervals corresponding to each mental activity.
  • Perception and Cognition Model: The system employs a two-phase approach:
  • Offline Phase: Brainwave features (phase relationship between left and right brain locations) are extracted and used to train a neuro-fuzzy network. This model learns the correlation between specific brainwave patterns and robot behaviors.
  • Online Phase: The trained model is used to decode the subject's current mental activity and command the robot to perform the corresponding walking behavior.
  • Signal Processing: Raw EEG signals are filtered to remove high-frequency noise and then decomposed into different frequency bands (delta, theta, alpha, beta) using band-pass or wavelet filters.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it describes the creation of a new dataset through experiments involving three human subjects. The data consists of:

  • 32-channel EEG recordings during imagined robot-walking behaviors.
  • Synchronized video recordings of subjects' hand postures.

The authors note that the brainwave patterns are subject-specific, implying that the "dataset" is effectively generated per individual for training the neuro-fuzzy model.

Key Insights and Ideas

  • Phase Relationship as a Key Feature: The study highlights the importance of the phase relationship between neural signals from different scalp locations (specifically T7 and T8, representing left and right brain activity) as a crucial feature for distinguishing between different imagined walking behaviors.
  • Subject-Specific Brainwave Signatures: A significant finding is that brainwave patterns correlated with the same mental activity (e.g., imagining "turning right") can look significantly different across individuals. This underscores the necessity of personalized training for BCI systems.
  • Hand Movement as a Correlate: Using simultaneous hand movements as a physical cue to help subjects focus on their mental imagery is an interesting approach to improve the reliability of brainwave acquisition for specific tasks.
  • Bridging Mental Imagery and Robot Action: The paper demonstrates a tangible link between a human's mental simulation of an action and the actual execution of that action by a humanoid robot, paving the way for more intuitive human-robot interaction.

Key Results

The paper presents a preliminary test of the BCI-based humanoid robot control system.

  • Successful Control of Basic Behaviors: The system was able to control the humanoid robot's walking behaviors (turning right, turning left, walking forward) based on the phase relationship of delta-band EEG signals from specific electrodes (T7 and T8).
  • Distinguishable Patterns: The study shows that the phase differences between brain signals at T7 and T8 are distinct for each of the three imagined walking behaviors:
  • "Turning right": T7 phase leads T8.
  • "Turning left": T8 phase leads T7.
  • "Walking forward": T7 and T8 phases are close to zero difference.
  • Qualitative Demonstration: Figure 6 visually demonstrates the robot executing a "turning left" behavior based on the brainwave control.

Limitations and Future Work

  • Muscular Signal Contamination: The authors acknowledge that recorded brainwaves might include muscular signals from hand movements, which could interfere with the identification of pure mental imagination.
  • Subject Concentration: The quality of recorded data is highly dependent on the subject's concentration on the mental activity.
  • Limited Behaviors: The current study focuses on only three basic walking behaviors.
  • Subject-Specific Training: The need for individual subject training for the neuro-fuzzy network is a practical limitation for widespread, immediate use.

Future Work:

  • Virtual Environment for Concentration: Developing a virtual environment to enhance subject concentration on mental activities.
  • Expanding Robot Capabilities: Controlling a wider range of robot behaviors, including shifting, arm movements, and head movements.
  • Comprehensive Algorithms: Investigating more sophisticated algorithms for recognizing human brain activities to afford more complex humanoid control.
  • Further Experimental Analysis: Presenting detailed analysis of experimental data in future work.
  • Planned Experiments: Conducting a number of experiments to further validate the proposed BCI-based humanoid robot system.

Summary updated

Abstract

The understanding of how humans process information, determine salience, and combine seemingly unrelated information is essential to automated processing of large amounts of information that is partially relevant, or of unknown relevance. Recent neurological science research in human perception, and in information science regarding contextbased modeling, provides us with a theoretical basis for using a bottom-up approach for automating the management of large amounts of information in ways directly useful for human operators. However, integration of human intelligence into a game theoretic framework for dynamic and adaptive decision support needs a perception and cognition model. For the purpose of cognitive modeling, we present a brain-computer-interface (BCI) based humanoid robot system to acquire brainwaves during human mental activities of imagining a humanoid robot-walking behavior. We use the neural signals to investigate relationships between complex humanoid robot behaviors and human mental activities for developing the perception and cognition model. The BCI system consists of a data acquisition unit with an electroencephalograph (EEG), a humanoid robot, and a charge couple CCD camera. An EEG electrode cup acquires brainwaves from the skin surface on scalp. The humanoid robot has 20 degrees of freedom (DOFs); 12 DOFs located on hips, knees, and ankles for humanoid robot walking, 6 DOFs on shoulders and arms for arms motion, and 2 DOFs for head yaw and pitch motion. The CCD camera takes video clips of the human subject's hand postures to identify mental activities that are correlated to the robot-walking behaviors. We use the neural signals to investigate relationships between complex humanoid robot behaviors and human mental activities for developing the perception and cognition model.
BibTeX
@inproceedings{li2012acquiring,
  author = {Wei Li and Yunyi Li and Genshe Chen and Dan Shen and Erik Blasch and Khanh Pham and Robert S. Lynch},
  title = {{Acquiring neural signals for developing a perception and cognition model}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2012},
  doi = {10.1117/12.918767},
}
Figure