Gradual Relation Network: Decoding Intuitive Upper Extremity Movement Imaginations Based on Few-Shot EEG Learning

Kyung-Hwan Shim, Ji-Hoon Jeong, Seong-Whan LeearXiv 2020 · OpenReview

TL;DR

This paper introduces the Gradual Relation Network (GRN), a novel few-shot learning approach for brain-computer interfaces (BCIs) that significantly reduces calibration time for decoding intuitive upper-extremity movement imaginations (MIs). The GRN demonstrates promising results in both offline classification and real-time robotic arm control, paving the way for more practical and user-friendly BCIs.

What Problem Does This Paper Address?

Traditional brain-computer interfaces (BCIs), particularly those used for controlling external devices like robotic arms, require extensive calibration for each user and session. This lengthy process is a major bottleneck, hindering real-world applications by consuming significant user time and potentially leading to fatigue and reduced performance. The paper aims to address this by developing a BCI system that can achieve robust decoding of intuitive movements with minimal training data, specifically focusing on upper-extremity movements.

Methods

The core innovation is the Gradual Relation Network (GRN), a metric-based few-shot learning model. The GRN is designed to:

  • Gradually combine temporal and spectral features: It processes EEG data by first extracting spectral and temporal information through convolutional layers. This information is then grouped into nine distinct sets.
  • Utilize a relation module: This module compares encoded features from the EEG data with "prototypical features" for each class. The comparison is done group by group, allowing the model to gradually build a comprehensive understanding of the user's intention.
  • Employ a few-shot learning approach: This allows the GRN to learn effectively from a small number of training samples (1-shot, 5-shot, and 25-shot settings), drastically reducing the need for extensive calibration data.
  • Focus on intuitive commands: Instead of relying solely on pre-defined classes, the GRN is designed to accommodate various intuitive movements related to specific body parts (upper-arm, forearm, hand), making the control more natural.

Datasets

The study utilized EEG data collected from 25 healthy, right-handed subjects performing upper-extremity movement imaginations (upper-arm, forearm, and hand). The data was acquired during controlled experimental sessions, with specific protocols for each body part. The paper also mentions using this data for both offline analysis and real-time robotic arm control experiments.

Key Insights and Ideas

  • Few-shot learning is a viable solution for BCI calibration: The paper strongly suggests that few-shot learning techniques can significantly reduce the time and effort required for BCI calibration, making BCIs more accessible.
  • Intuitive control is achievable with minimal data: By focusing on body part sub-groups and a metric-based approach, the GRN can decode user intentions for a range of movements without needing explicit training for every single command.
  • Gradual feature combination enhances robustness: The GRN's ability to progressively combine spectral and temporal features across different groups helps compensate for the low signal-to-noise ratio (SNR) inherent in EEG data.
  • Real-time robotic arm control is feasible: The successful demonstration of the GRN in controlling a robotic arm for a drinking task highlights the practical implications of this research.

Key Results

  • Offline Classification: The GRN achieved grand average multi-class classification accuracies of 42.57% (1-shot), 55.60% (5-shot), and 80.85% (25-shot) across all subjects and sessions.
  • Candidate Class Performance: The GRN also showed strong performance on untrained "candidate" classes, indicating its ability to generalize to intuitive commands not explicitly trained.
  • Real-time Robotic Arm Control: In an online experiment, five participants successfully controlled a robotic arm for a drinking task with a success rate of 78%. This was achieved with a significantly shortened calibration time (10 trials per representative class).

Limitations and Future Work

  • Real-time Asynchronous Control: The paper acknowledges that while the GRN shows promise for intuitive control, its current performance with five training trials is not yet sufficient for realizing safe asynchronous robotic arm control. Future work should focus on improving this aspect.
  • Complex Multi-part Movements: The current GRN design primarily focuses on sub-parts of the upper extremity. Future research should explore decoding more complex, simultaneous movements involving multiple sub-parts (e.g., hand grasping while arm reaching).
  • Alternative Few-Shot Architectures: The authors suggest exploring other few-shot learning approaches, such as recurrent neural network (RNN) memory-based methods and fine-tuning techniques, which might be more appropriate for BCI domains than the current metric-based GRN.

Summary updated

Abstract

Brain-computer interface (BCI) is a communication tool that connects users and external devices. In a real-time BCI environment, a calibration procedure is particularly necessary for each user and each session. This procedure consumes a significant amount of time that hinders the application of a BCI system in a real-world scenario. To avoid this problem, we adopt the metric based few-shot learning approach for decoding intuitive upper-extremity movement imagination (MI) using a gradual relation network (GRN) that can gradually consider the combination of temporal and spectral groups. We acquired the MI data of the upper-arm, forearm, and hand associated with intuitive upper-extremity movement from 25 subjects. The grand average multiclass classification results under offline analysis were 42.57%, 55.60%, and 80.85% in 1-, 5-, and 25-shot settings, respectively. In addition, we could demonstrate the feasibility of intuitive MI decoding using the few-shot approach in real-time robotic arm control scenarios. Five participants could achieve a success rate of 78% in the drinking task. Hence, we demonstrated the feasibility of the online robotic arm control with shortened calibration time by focusing on human body parts but also the accommodation of various untrained intuitive MI decoding based on the proposed GRN.
BibTeX
@misc{shim2020gradual,
  author = {Kyung-Hwan Shim and Ji-Hoon Jeong and Seong-Whan Lee},
  title = {{Gradual Relation Network: Decoding Intuitive Upper Extremity Movement Imaginations Based on Few-Shot EEG Learning}},
  year = {2020},
  eprint = {2005.02602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.02602},
}
Figure