Decoding of Intuitive Visual Motion Imagery Using Convolutional Neural Network under 3D-BCI Training Environment

Byoung-Hee Kwon, Ji-Hoon Jeong, Jeong-Hyun Cho, Seong-Whan LeearXiv 2020 · OpenReview

TL;DR

This paper introduces a novel 3D virtual reality BCI training platform to enhance intuitive visual motion imagery (VMI) for decoding user intentions. By combining functional connectivity for channel selection with a custom Convolutional Neural Network (CNN), they achieved a promising 67.5% average classification accuracy for four distinct daily-life movements, paving the way for more natural BCI control.

What Problem Does This Paper Address?

Traditional Brain-Computer Interfaces (BCIs) often rely on motor imagery (MI), where users imagine muscle movements. While effective, this can be mentally taxing, lead to user fatigue, and suffer from inter-subject variability in brain signal patterns, requiring extensive calibration. This paper addresses the need for a more intuitive and less demanding BCI paradigm. They propose using visual motion imagery (VMI) – imagining visual representations of movements – as a more accessible alternative to MI. The challenge lies in effectively decoding these VMI signals, especially given their subtle nature and the inherent noise in EEG data. Furthermore, existing BCI systems often require a large number of EEG channels, increasing complexity and cost.

Methods

The core of this research lies in two key innovations:

  • 3D Virtual Reality BCI Training Platform: To facilitate intuitive VMI, the authors developed a 3D virtual reality environment. This platform presents users with visual cues of common daily actions (picking up a phone, opening a door, eating, pouring water) in a realistic 3D space. This immersive approach aims to make the imagination process more natural and less prone to the inconsistencies of purely mental motor imagery.
  • Functional Connectivity for Channel Selection & CNN Architecture: After collecting EEG data during VMI tasks, the researchers employed a functional connectivity approach. This method analyzes the correlation between different EEG channels to identify those most relevant to VMI. By focusing on channels with high correlation (above 0.9) primarily in the prefrontal and occipital lobes, they significantly reduced the number of channels needed for effective decoding. This reduced channel set was then fed into a custom-designed Convolutional Neural Network (CNN). The CNN architecture, featuring multiple convolutional and pooling layers, was specifically designed to extract both spatial and temporal features from the EEG data for robust classification of the imagined movements.

Datasets

The study utilized a dataset collected from nine healthy subjects (ages 24-30, 5 male, 4 female, all right-handed) who were naive to BCI experiments. The data was acquired using 64 Ag/AgCl electrodes at a 1000 Hz sampling rate. For each subject, the experiment consisted of 200 trials, with 50 trials per of the four VMI classes (picking up a cell phone, opening a door, eating food, pouring water) and a resting state. This constitutes a newly introduced benchmark dataset for visual motion imagery decoding within their developed 3D BCI training environment.

Key Insights and Ideas

  • VMI as a Viable, Intuitive BCI Paradigm: The study strongly suggests that visual motion imagery can serve as a more intuitive and less fatiguing alternative to traditional motor imagery for BCI control. The 3D VR environment appears to enhance the effectiveness of this paradigm.
  • Functional Connectivity is Key for Channel Reduction: Identifying relevant EEG channels using functional connectivity significantly improves decoding performance and reduces system complexity. This approach effectively filters out noise and focuses on brain regions most active during VMI.
  • CNNs Excel at VMI Decoding: The proposed CNN architecture, when combined with the selected channels, demonstrates strong performance in classifying VMI, outperforming conventional methods like CSP-LDA and deepConvNet in certain configurations.
  • Prefrontal and Occipital Lobes are Crucial: Statistical analysis confirms that visual motion imagery primarily activates the prefrontal and occipital lobes, aligning with the findings from functional connectivity analysis.

Key Results

The primary quantitative result is the average classification performance of 67.50% (±1.52%) for the proposed CNN architecture using 16 selected channels across all nine subjects for four distinct classes. This is a significant improvement over conventional methods like CSP-LDA (30.97%) and deepConvNet (63.47%) when evaluated on the same dataset with 64 channels. Notably, the performance with 16 channels using the proposed architecture (67.50%) surpassed deepConvNet's performance with 64 channels. The study also found that VMI signals showed significant activation in the [0.5-13] Hz frequency band approximately 500 ms after the visual cue.

Limitations and Future Work

Limitations:

  • The study acknowledges that while the proposed architecture shows high performance, overcoming the "lack of training set" remains a challenge. This implies that the dataset size, while sufficient for this study, might be a bottleneck for more complex scenarios or for achieving even higher accuracies.
  • The paper mentions that the proposed architecture, with its current number of convolutional layers, might not fully capture all temporal information.

Future Work:

  • Data Augmentation: The authors plan to address the limited training data by exploring various data augmentation methods.
  • Deeper Neural Networks: To better capture temporal dynamics, they intend to investigate deeper neural network architectures.
  • Multi-Command Control: Future work will focus on expanding the system to handle a greater number of commands, enabling higher degrees of freedom for BCI control.
  • Real-World Applications: The ultimate goal is to contribute to the development of BCI-based devices like robotic arms and neuroprosthetics, providing a more convenient and intuitive human-machine interaction.

Summary updated

Abstract

In this study, we adopted visual motion imagery, which is a more intuitive brain-computer interface (BCI) paradigm, for decoding the intuitive user intention. We developed a 3-dimensional BCI training platform and applied it to assist the user in performing more intuitive imagination in the visual motion imagery experiment. The experimental tasks were selected based on the movements that we commonly used in daily life, such as picking up a phone, opening a door, eating food, and pouring water. Nine subjects participated in our experiment. We presented statistical evidence that visual motion imagery has a high correlation from the prefrontal and occipital lobes. In addition, we selected the most appropriate electroencephalography channels using a functional connectivity approach for visual motion imagery decoding and proposed a convolutional neural network architecture for classification. As a result, the averaged classification performance of the proposed architecture for 4 classes from 16 channels was 67.50 % across all subjects. This result is encouraging, and it shows the possibility of developing a BCI-based device control system for practical applications such as neuroprosthesis and a robotic arm.
BibTeX
@misc{kwon2020decoding,
  author = {Byoung-Hee Kwon and Ji-Hoon Jeong and Jeong-Hyun Cho and Seong-Whan Lee},
  title = {{Decoding of Intuitive Visual Motion Imagery Using Convolutional Neural Network under 3D-BCI Training Environment}},
  year = {2020},
  eprint = {2005.08879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.08879},
}
Figure