BELT: Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision

Jinzhao Zhou, Yiqun Duan, Yu-Cheng Chang, Yu-Kai Wang, Chin-Teng Lin

TL;DR

This paper introduces BELT, a novel framework that leverages large language models (LLMs) to bootstrap the learning of electroencephalography (EEG) representations for brain-to-language translation and zero-shot sentiment classification. By using natural language as supervision, BELT significantly improves decoding accuracy on limited EEG datasets, paving the way for more sophisticated brain-computer interfaces.

What Problem Does This Paper Address?

Decoding brain signals, particularly non-invasive EEG, into natural language is a long-standing challenge with immense potential for brain-computer interfaces (BCIs). A major hurdle is the scarcity of large, high-quality, and semantically diverse EEG datasets, which are crucial for training robust deep learning models. Existing methods often struggle with limited vocabularies and generalization to unseen data. This paper aims to bridge this gap by using the rich semantic understanding of pre-trained LLMs to guide the learning of EEG representations, enabling more effective brain-to-language translation and zero-shot sentiment analysis.

Methods

The core of BELT is a novel learning framework that bootstraps EEG representation learning using off-the-shelf LLMs. Key components include:

  • Conformer Encoder: A conformer network is employed to process segmented EEG signals, capturing both local and global dependencies crucial for EEG analysis.
  • Vector Quantization (VQ): The EEG representations are quantized into a discrete codebook. This process enhances robustness by making representations invariant to minor perturbations and retaining only essential information.
  • Contrastive Learning with Natural Language Supervision: This is the central innovation. BELT aligns the EEG representation space with that of a pre-trained LLM. It uses word embeddings from the LLM as supervision, contrasting positive EEG-word pairs with negative ones to learn semantically meaningful EEG representations. This allows the model to learn from the LLM's vast knowledge without requiring massive EEG datasets.
  • Translation Finetuning: A pre-trained BART model is finetuned for the brain-to-language translation task, taking the learned EEG representations as input.
  • Zero-Shot Sentiment Classification: The learned EEG representations are also used for sentiment classification. This is framed as a decoder-classifier task, where the model learns to predict sentiment labels from EEG signals without explicit sentiment labels during training for the EEG data itself.

Datasets

  • ZuCo Dataset: This dataset was used for training and evaluating the brain-to-language translation and zero-shot sentiment classification tasks. It contains EEG data recorded during natural reading tasks, along with eye-tracking data for segmentation. The dataset includes both normal reading and task-specific reading materials, with sentiment labels provided for the latter.
  • Stanford Sentiment Treebank: This dataset was used for training the language model component for sentiment classification.

Key Insights and Ideas

  • LLMs as a "Semantic Anchor" for EEG: The paper demonstrates that LLMs can effectively "bootstrap" the learning of EEG representations. By aligning EEG signals with language embeddings, the model gains a richer understanding of semantic content, overcoming the limitations of small EEG datasets.
  • Natural Language Supervision for Data Efficiency: The core idea of using natural language supervision is a significant step towards making BCI research more data-efficient. It leverages existing, massive text corpora to compensate for the inherent difficulty in collecting large-scale, annotated EEG data.
  • Vector Quantization Enhances Robustness: Quantizing EEG representations proves beneficial, making the model more resilient to noise and variations in the EEG signals.
  • Open Vocabulary Decoding: BELT moves beyond limited vocabularies, enabling the decoding of more complex thoughts and sentences, which is crucial for practical BCI applications.

Key Results

  • Brain-to-Language Translation: BELT achieved state-of-the-art results, outperforming baseline models by 5.45% on BLEU-1 scores. Specifically, it reached a BLEU-1 score of 42.31% and improved upon previous methods by significant margins across various BLEU-N metrics.
  • Zero-Shot Sentiment Classification: BELT achieved 67.32% accuracy on the zero-shot sentiment classification task, significantly outperforming traditional end-to-end methods and achieving competitive results with larger language models.
  • Qualitative Improvements: The model demonstrated an improved ability to decode verbs and nouns, suggesting potential for more precise control applications. It also showed promise in decoding critical concepts, hinting at future capabilities for conveying complex ideas.

Limitations and Future Work

  • Decoding Accuracy: While BELT shows significant improvements, the decoding results are still substantially worse than those achieved in language translation and automatic speech recognition.
  • Shallow vs. Deep Representation: For translation, deeper language representations did not significantly impact performance, while for sentiment classification, they tended to be negative. This suggests that for word-level tasks, shallower representations might be more beneficial, while deeper ones might introduce too much sentence-specific context that can confuse the EEG encoder.
  • Future Work:
    • Integrating larger or more advanced LLMs into the framework.
  • Collecting more language-related EEG data.
  • Exploring more efficient text-summary-based pipelines instead of direct text translation.
  • Investigating the impact of different contrastive learning strategies (inter- vs. intra-modality) and their combinations.
  • Further exploring the trade-offs between model complexity, contrastive coefficient, and performance.

Summary updated

Abstract

This paper presents BELT, a novel model and learning framework for the pivotal topic of brain-to-language translation research. The translation from noninvasive brain signals into readable natural language has the potential to promote the application scenario as well as the development of brain-computer interfaces (BCI) as a whole. The critical problem in brain signal decoding or brain-to-language translation is the acquisition of semantically appropriate and discriminative EEG representation from a dataset of limited scale and quality. The proposed BELT method is a generic and efficient framework that bootstraps EEG representation learning using off-the-shelf large-scale pretrained language models (LMs). With a large LM's capacity for understanding semantic information and zero-shot generalization, BELT utilizes large LMs trained on Internet-scale datasets to bring significant improvements to the understanding of EEG signals. In particular, the BELT model is composed of a deep conformer encoder and a vector quantization encoder. Semantical EEG representation is achieved by a contrastive learning step that provides natural language supervision. We achieve state-of-the-art results on two featuring brain decoding tasks including the brain-to-language translation and zero-shot sentiment classification. Specifically, our model surpasses the baseline model on both tasks by 5.45% and over 10% and archives a 42.31% BLEU-1 score and 67.32% precision on the main evaluation metrics for translation and zero-shot sentiment classification respectively.
BibTeX
@article{zhou2024belt,
  author = {Jinzhao Zhou and Yiqun Duan and Yu-Cheng Chang and Yu-Kai Wang and Chin-Teng Lin},
  title = {{BELT: Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision}},
  journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  year = {2024},
  doi = {10.1109/tnsre.2024.3450795},
}
Figure