A Characterization of Brain-Computer Interface Performance Trade-Offs Using Support Vector Machines and Deep Neural Networks to Decode Movement Intent

Nicholas Skomrock, Michael A. Schwemmer, Jordyn E. Ting, Hemang R. Trivedi, Gaurav Sharma, Marcia Bockbrader, David A. FriedenbergFrontiers in Neuroscience 2018 · DOI

TL;DR

This paper investigates the critical trade-offs between accuracy, response time, and the number of controllable functions in brain-computer interfaces (BCIs) for decoding movement intent. By comparing Support Vector Machines (SVMs) and Deep Neural Networks (DNNs) using intracortical recordings from a participant with tetraplegia, the study reveals that DNNs offer superior performance, especially as the complexity of movements increases, and can be effectively used in real-time to control functional electrical stimulation (FES).

What Problem Does This Paper Address?

Brain-computer interfaces (BCIs) hold immense promise for restoring function to individuals with paralysis by translating neural activity into control signals for assistive devices. However, for these systems to transition from the lab to widespread clinical use, they must meet user-defined performance criteria. Surveys consistently highlight three key features: high accuracy, fast response times, and the ability to control multiple functions (multi-functionality). The challenge lies in the inherent conflict between these goals: optimizing for one often degrades the others. For instance, increasing the number of decodable movements can decrease accuracy and slow down response times. Despite the importance of these trade-offs, there's a lack of quantitative understanding of how different decoding strategies and design choices impact these critical performance metrics simultaneously. This paper aims to fill that gap by systematically characterizing these trade-offs.

Methods

The study employed two primary decoding algorithms:

  • Support Vector Machine (SVM): This is a well-established, state-of-the-art classifier for discrete movement decoding in laboratory settings. The SVM used here was trained daily on a single block of data and evaluated on a subsequent block.
  • Deep Neural Network (DNN): A novel DNN framework was developed and evaluated. This network processes a sequence of neural features (96 channels over 900 ms) to predict movement intent.

The researchers also investigated the impact of a specific data preprocessing step: a 1-second boxcar filter used to smooth neural features. To quantify performance trade-offs across varying complexity, they created synthetic datasets by combining and subsampling data from two- and four-movement tasks to simulate scenarios with one to six movements. Finally, a real-time demonstration was conducted where the DNN decoder controlled functional electrical stimulation (FES) of the participant's forearm.

Datasets

The primary dataset used was historical intracortical recordings from a single study participant with a C5 AIS category A traumatic spinal cord injury. This participant imagined performing various hand and finger movements.

  • Offline Analysis Datasets:
    • Two-movement task: Imagined hand open/close.
  • Four-movement task: Imagined index extension, index flexion, wrist extension, and wrist flexion.
  • Synthetic Datasets: Created by combining and subsampling the two- and four-movement data to simulate tasks with 1, 2, 3, 4, 5, or 6 movements.
  • Real-Time Demonstration Dataset: Data collected during the participant's real-time control of FES.
  • Able-Bodied Benchmark Dataset: Data collected from three able-bodied participants performing the same six-movement task to establish a baseline for ideal response times.

Key Insights and Ideas

  • DNNs Outperform SVMs with Increasing Complexity: The most striking finding is that as the number of decodable movements increases, DNNs maintain higher accuracy and faster response times compared to SVMs. This suggests DNNs are more scalable for complex BCI applications.
  • Preprocessing Trade-offs Differ: The 1-second boxcar filter, crucial for SVM performance, had little impact on the DNN's success rate. However, it did increase the DNN's response time, highlighting that preprocessing choices have distinct effects on different decoder architectures.
  • Real-time Control Achieved: The study successfully demonstrated real-time control of FES using the DNN, allowing the participant to perform six distinct hand/finger movements, bridging the gap between offline analysis and practical application.
  • Quantifying the "Ideal" Target: By comparing the participant's performance with able-bodied individuals, the study provides a quantitative benchmark for ideal BCI-FES performance, offering a valuable target for future development.

Key Results

  • Accuracy and Response Time Degradation: For both SVM and DNN, accuracy decreased and response time increased as the number of movements increased.
  • DNN Advantage: The decline in accuracy and increase in response time were significantly less pronounced for the DNN compared to the SVM.
  • Accuracy: DNN accuracy was consistently high (e.g., ~97% for 4 movements), while SVM accuracy dropped significantly (e.g., ~60% for 4 movements with boxcar filter).
  • Response Time: DNN response times were faster (e.g., 506-524 ms for 2-4 movements without boxcar) than SVM response times (e.g., 855-946 ms for 2-4 movements with boxcar).
  • Preprocessing Impact:
    • SVM success rate dropped from ~60% to ~12.7% without the boxcar filter.
  • DNN success rate was largely unaffected by the boxcar filter.
  • The boxcar filter increased DNN response time from ~524 ms to ~719 ms.
  • Real-time FES Control: The participant achieved a 76.4% success rate in the real-time FES task.
  • Response Time Comparison: The tetraplegic participant's average response time (997 ± 224 ms) was significantly slower than able-bodied participants (390 ± 96 ms), with a ~607 ms difference. The decoder itself contributed ~821 ms of latency.

Limitations and Future Work

  • Single Participant Study: The study relied on data from a single participant with tetraplegia. Future work needs to explore how these findings generalize to other individuals, considering the invasiveness of intracortical BCIs.
  • Task and Decoder Specificity: The reported trade-offs are specific to SVM and DNN decoders for discrete hand movements from intracortical data. Different tasks, decoders (e.g., EEG-based), or modalities might exhibit different trade-offs.
  • Offline vs. Online Performance: While the real-time demonstration showed promise, further work is needed to confirm if the observed offline trade-offs carry over to online scenarios with visual feedback.
  • Joint Optimization: Current model training primarily optimizes for accuracy. Future research should explore methods for jointly optimizing multiple performance characteristics (accuracy, response time, number of functions) directly during decoder training, potentially using weighted averages of objective functions.
  • User Input: The study acknowledges that incorporating direct user feedback on performance criteria is crucial for clinical adoption, as actual user experiences might differ from survey-based expectations.

Summary updated

Abstract

Laboratory demonstrations of brain-computer interface (BCI) systems show promise for reducing disability associated with paralysis by directly linking neural activity to the control of assistive devices. Surveys of potential users have revealed several key BCI performance criteria for clinical translation of such a system. Of these criteria, high accuracy, short response latencies, and multi-functionality are three key characteristics directly impacted by the neural decoding component of the BCI system, the algorithm that translates neural activity into control signals. Building a decoder that simultaneously addresses these three criteria is complicated because optimizing for one criterion may lead to undesirable changes in the other criteria. Unfortunately, there has been little work to date to quantify how decoder design simultaneously affects these performance characteristics. Here, we systematically explore the trade-off between accuracy, response latency, and multi-functionality for discrete movement classification using two different decoding strategies-a support vector machine (SVM) classifier which represents the current state-of-the-art for discrete movement classification in laboratory demonstrations and a proposed deep neural network (DNN) framework. We utilized historical intracortical recordings from a human tetraplegic study participant, who imagined performing several different hand and finger movements. For both decoders, we found that response time increases (i.e., slower reaction) and accuracy decreases as the number of functions increases. However, we also found that both the increase of response times and the decline in accuracy with additional functions is less for the DNN than the SVM. We also show that data preprocessing steps can affect the performance characteristics of the two decoders in drastically different ways. Finally, we evaluated the performance of our tetraplegic participant using the DNN decoder in real-time to control functional electrical stimulation (FES) of his paralyzed forearm. We compared his performance to that of able-bodied participants performing the same task, establishing a quantitative target for ideal BCI-FES performance on this task. Cumulatively, these results help quantify BCI decoder performance characteristics relevant to potential users and the complex interactions between them.
BibTeX
@inproceedings{skomrock2018characterization,
  author = {Nicholas Skomrock and Michael A. Schwemmer and Jordyn E. Ting and Hemang R. Trivedi and Gaurav Sharma and Marcia Bockbrader and David A. Friedenberg},
  title = {{A Characterization of Brain-Computer Interface Performance Trade-Offs Using Support Vector Machines and Deep Neural Networks to Decode Movement Intent}},
  booktitle = {Frontiers in Neuroscience},
  year = {2018},
  doi = {10.3389/fnins.2018.00763},
}
Figure