Rethinking the Methods and Algorithms for Inner Speech Decoding and Making Them Reproducible

Foteini Liwicki, Vibha Gupta, Rajkumar Saini, Kanjar De, Marcus Liwicki

TL;DR

This paper tackles the challenge of decoding "inner speech" – the silent voice in our heads – using non-invasive EEG. By meticulously investigating preprocessing techniques and a custom CNN architecture, the authors achieve state-of-the-art results on a public dataset for classifying vowels and words, while crucially emphasizing reproducibility through open-source code and data.

What Problem Does This Paper Address?

Inner speech, the internal monologue we experience when thinking, reading, or planning, has long been a subject of philosophical and psychological inquiry. More recently, researchers have begun exploring its potential for brain-computer interfaces (BCIs), aiming to decode these silent thoughts into actionable commands or even synthesized speech. However, a significant hurdle in this field is the lack of reproducibility. Many studies suffer from limited data availability, proprietary code, and inconsistent methodologies, making it difficult to build upon existing work. This paper directly addresses this by not only proposing improved methods for inner speech decoding but also by making their entire workflow, including code and processed data, publicly available.

Methods

The core of this research lies in a deep Convolutional Neural Network (CNN) architecture, dubbed "iSpeech-CNN," adapted for inner speech decoding. The authors explored several key methodological aspects:

  • Preprocessing Pipeline: They investigated the impact of various preprocessing steps, including bandpass filtering (2-40 Hz), down-sampling, and Independent Component Analysis (ICA) for artifact removal. Notably, they found that down-sampling actually degraded performance, leading them to exclude it from their final architecture.
  • CNN Architecture: The iSpeech-CNN is based on the EEGNet architecture but with modifications to the number and size of filters in the initial convolutional layers. They found that increasing filters in the early layers (from 20 to 40) improved performance.
  • Experimental Approaches: The study compared three main approaches:
  • Subject-Dependent (Within-Subject): Training a separate model for each individual subject.
  • Subject-Independent (Leave-One-Out): Training a model on all subjects except one, then testing on the excluded subject.
  • Mixed Approach: Training a single model on data from all subjects.

Datasets

The study utilized a publicly available Electroencephalography (EEG) dataset from [8]. This dataset contains recordings from 15 subjects performing inner and outer speech tasks for five vowels (/a/, /e/, /i/, /o/, /u/) and six words (arriba/up, abajo/down, derecha/right, izquierda/left, adelante/forward, and atrás/backwards). The experiment involved a ready interval, a stimulus presentation for inner speech production, and a rest interval. The authors specifically focused on the inner speech portion of the experiment.

Key Insights and Ideas

  • Reproducibility is Paramount: The paper strongly advocates for and demonstrates the importance of making code and data publicly available to foster scientific progress in this challenging field.
  • Down-sampling Hurts: Counterintuitively, down-sampling the EEG signals led to a loss of information and reduced classification accuracy, highlighting the need for careful consideration of sampling rates in EEG analysis.
  • Preprocessing Matters: The choice of preprocessing steps, particularly ICA for artifact removal, significantly impacts performance.
  • Architecture Tuning is Crucial: Modifying the number of filters in the initial layers of the CNN architecture led to notable performance improvements.
  • Subject-Dependent Models Excel: For inner speech decoding with EEG, subject-dependent models generally outperformed subject-independent or mixed approaches, suggesting that individual brain patterns are highly variable.

Key Results

The study achieved state-of-the-art performance on the chosen dataset:

  • Vowel Classification: A mean accuracy of 35.20% was achieved for classifying five vowels using the iSpeech-CNN architecture without down-sampling.
  • Word Classification: A mean accuracy of 29.21% was achieved for classifying six words using the iSpeech-CNN architecture without down-sampling.

These results represent an improvement over previous work on the same dataset, particularly in the word classification task. The paper also provides detailed performance metrics (precision, weighted F-score, F-score) in comparison to prior studies.

Limitations and Future Work

  • Generalizability: The authors acknowledge that generalizing performance across all subjects remains a significant challenge in EEG-based inner speech decoding. They note that even with the mixed approach, accuracies were largely random.
  • Limited Architectures Explored: The study focused on CNN architectures, specifically building upon the work of Cooney et al. [7]. Other deep learning architectures were not investigated.
  • Future Directions: The authors suggest future work will involve evaluating their system on other publicly available datasets and further investigating optimal frequency sub-bands and down-sampling rates specific to individual subjects. They also plan to explore directions that could lead to more generalizable models.

Summary updated

Abstract

This study focuses on the automatic decoding of inner speech using noninvasive methods, such as Electroencephalography (EEG). While inner speech has been a research topic in philosophy and psychology for half a century, recent attempts have been made to decode nonvoiced spoken words by using various brain-computer interfaces. The main shortcomings of existing work are reproducibility and the availability of data and code. In this work, we investigate various methods (using Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Long Short-Term Memory Networks (LSTM)) for the detection task of five vowels and six words on a publicly available EEG dataset. The main contributions of this work are (1) subject dependent vs. subject-independent approaches, (2) the effect of different preprocessing steps (Independent Component Analysis (ICA), down-sampling and filtering), and (3) word classification (where we achieve state-of-the-art performance on a publicly available dataset). Overall we achieve a performance accuracy of 35.20% and 29.21% when classifying five vowels and six words, respectively, in a publicly available dataset, using our tuned iSpeech-CNN architecture. All of our code and processed data are publicly available to ensure reproducibility. As such, this work contributes to a deeper understanding and reproducibility of experiments in the area of inner speech detection.
BibTeX
@inproceedings{liwicki2022rethinking,
  author = {Foteini Liwicki and Vibha Gupta and Rajkumar Saini and Kanjar De and Marcus Liwicki},
  title = {{Rethinking the Methods and Algorithms for Inner Speech Decoding and Making Them Reproducible}},
  booktitle = {NeuroSci},
  year = {2022},
  doi = {10.3390/neurosci3020017},
}
Figure