Real-Time Navigation in Google Street View® Using a Motor Imagery-Based BCI

Liuyin Yang, Marc M. Van HulleSensors 2023 · DOI

TL;DR

This paper presents a novel Brain-Computer Interface (BCI) system for real-time navigation in Google Street View® using motor imagery (MI) with dry EEG electrodes. The system addresses the challenges of low signal quality from dry electrodes and the inherent difficulty of decoding MI by introducing a middle-level control scheme and eye-blink-based error correction, achieving over 80% success rates in online experiments.

What Problem Does This Paper Address?

Traditional virtual reality (VR) and game navigation heavily relies on external controllers. Brain-Computer Interfaces (BCIs) offer an alternative, particularly for individuals with motor impairments, by translating brain activity into commands. However, most BCI navigation systems use cue-based visual paradigms (like P300 or SSVEP), which can be less intuitive and comfortable than motor imagery (MI). MI-based BCIs, while more natural, face significant challenges: decoding MI from EEG is difficult, especially with dry electrodes that offer lower signal-to-noise ratios (SNRs) and are typically used in fewer numbers. Furthermore, existing control schemes for BCI navigation are often either too low-level (requiring fine control, leading to inefficiency) or too high-level (limiting user agency to predefined destinations). This paper aims to bridge this gap by developing an intuitive, user-friendly MI-BCI for immersive VR navigation that overcomes the limitations of dry electrodes and offers a more flexible control strategy for unknown virtual environments.

Methods

The core of the system is a motor imagery (MI) BCI that uses an 8-channel dry EEG headset. Participants imagine specific movements (two-hand clenching for forward movement, left/right hand clenching for turning) to control navigation in Google Street View®.

Key methodological contributions include:

  • Dry Electrode EEG: Utilizing an 8-channel dry electrode wireless headset for convenience and ease of use, despite the inherent SNR challenges.
  • Motor Imagery Paradigm: Employing a self-paced, two-hand clenching action for starting movement and a cued-based, shorter clenching action for left/right turns.
  • Middle-Level Control Scheme: This novel approach aims for a balance between user control and efficiency. It allows users to initiate movement and then stop at junctions, where they can choose to turn or continue forward. This is more flexible than low-level control and more interactive than high-level control.
  • Eye Blink Error Correction: To mitigate decoding inaccuracies, eye blinks are integrated as an error-control signal. This allows users to correct erroneous navigation commands or switch between navigation modes. Two specific error-control strategies were implemented: Double Confirmation (DC) and Error Correction (EC).
  • FBCSP Decoder: A Filter Bank Common Spatial Pattern (FBCSP) approach is used for feature extraction, followed by a Bidirectional Long Short-Term Memory (BiLSTM) network for classification. This architecture is designed to handle the temporal dependencies in EEG data.

Datasets

The study used data collected from 21 healthy adult participants.

  • Offline Training and Testing: Participants underwent offline training sessions where their EEG data was recorded while performing MI tasks. This data was used to train and test individual MI decoders.
  • Online Experiments: Participants then used their trained decoders in real-time to navigate Google Street View®. The performance was evaluated based on task completion, accuracy, and time.

No pre-existing public datasets were explicitly mentioned as being used for training or evaluation. The study generated its own dataset through experiments with human participants.

Key Insights and Ideas

  • System Design is Crucial for Low-SNR BCIs: The paper strongly emphasizes that even with imperfect decoder accuracy (due to dry electrodes and limited channels), a well-designed BCI system (control logic, error correction) can still achieve functional and usable navigation.
  • Middle-Level Control Offers a Sweet Spot: The proposed middle-level control scheme proved effective for navigating unknown virtual environments, allowing users to intervene at key points (junctions) without needing constant fine-grained control.
  • Eye Blinks as a Versatile Control Signal: Integrating eye blinks not only as an artifact but as a deliberate control signal for error correction and mode switching significantly enhances the robustness and usability of the MI-BCI.
  • Error Correction Strategies Matter: The comparison between different error-control strategies (DC vs. EC) highlighted the importance of efficient error handling. The EC strategy, in particular, showed promising results in reducing navigation mistakes with an acceptable increase in control time.
  • MI-BCI for Immersive VR is Feasible: Despite the challenges, the study demonstrates that MI-BCI can be a viable option for controlling navigation in immersive VR environments like Google Street View®, paving the way for more natural and accessible VR experiences.

Key Results

  • Decoder Performance: Offline, the flexion-rest decoder achieved an average accuracy of approximately 80%, while the left-right decoder averaged around 63%. Online, accuracies showed greater variability across subjects but achieved similar performance levels.
  • Online Navigation Success Rate: The overall success rate for online experiments was over 80%, with the Middle-Level-DC strategy achieving 0.882 and Middle-Level-EC achieving 0.857. The "No Error Control" strategy had a significantly lower success rate (0.50).
  • Error Control Impact: The EC strategy was faster than the DC strategy for issuing navigation commands. The number of navigation errors was significantly reduced with error-control strategies compared to no error control.
  • User Experience: While not formally surveyed for usability, user feedback indicated that participants struggled with the system and often needed guidance, especially when the system made mistakes. This suggests that while technically feasible, user experience and training remain critical areas for improvement.

Limitations and Future Work

  • Google Street View® Limitations: The system is constrained by Google Street View®'s inherent navigation path limitations, which restrict user freedom.
  • Junction Detection Inconsistencies: Inconsistencies between Google Maps and GeoNames Web Service for junction information sometimes led to inaccurate stopping points.
  • Decoder Accuracy: The decoder accuracies, while improved by system design, are still lower than those achieved with visual paradigms like P300 or SSVEP.
  • Healthy Participants Only: The study was conducted on healthy individuals. Future work should include paralyzed individuals to assess real-world applicability.
  • Usability Testing: More detailed and quantitative usability tests are needed to fully understand user experience and identify areas for improvement.
  • Algorithm Development: Further research into new MI decoding algorithms tailored for dry electrodes and low SNR is recommended.
  • Passive Error Detection: Exploring passive error detection mechanisms (e.g., error-related potentials) could further enhance system design.
  • Real-World Application: Adapting the middle-level control pattern to real-world navigation scenarios, leveraging advances in computer vision and autonomous driving, is a promising direction.

Summary updated

BibTeX
@article{yang2023real,
  author = {Liuyin Yang and Marc M. Van Hulle},
  title = {{Real-Time Navigation in Google Street View® Using a Motor Imagery-Based BCI}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23031704},
}
Figure