Can Brain Signals Reveal Inner Alignment with Human Languages?

William Han, Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Douglas Weber, Bo Li, Ding Zhao

TL;DR

Researchers have developed MTAM (Multimodal Transformer Alignment Model), a novel framework that aligns human brain activity (EEG) with natural language processing. By using hierarchical transformers and specialized alignment losses like Wasserstein Distance, the model achieves new state-of-the-art results in sentiment analysis and relation detection while providing fascinating neuroscientific insights into how our brains map specific words to emotional states.

What Problem Does This Paper Address?

While modern Large Language Models (LLMs) like GPT-4 are incredibly capable, they are often criticized for being "data-hungry" and fundamentally different from how the human brain processes language. Humans understand linguistic rules effortlessly, yet the connection between our internal cognitive state—measured via Electroencephalography (EEG)—and the external representation of language hasn't been deeply explored in a computational multimodal context.

The authors aim to bridge this gap. They ask: Can we improve machine understanding of language by aligning it with the way the human brain encodes it? By finding the "inner alignment" between these two modalities, the researchers hope to both boost AI performance on downstream tasks and gain a clearer physiological understanding of language processing in the brain.

Methods

The core of the study is the Multimodal Transformer Alignment Model (MTAM). It employs a dual-encoder (or "bi-encoder") architecture to project EEG signals and text into a shared vector space.

  1. Hierarchical Transformer Encoders: The model uses separate transformer-based encoders for EEG and text. The EEG encoder treats brain signals as low-level embeddings, while the text encoder uses a pre-trained LLM (like BERT) to extract initial features before passing them through its own transformer layers.
  2. Cross Alignment Module (CAM): This is the "secret sauce." Instead of just concatenating features, the authors introduced a module that uses two specific relationship-seeking loss functions:
  3. Canonical Correlation Analysis (CCA): Maximizes the linear correlation between the two modalities.
  4. Wasserstein Distance (WD): A metric from Optimal Transport that aligns distributions by accounting for the underlying geometry of the feature space.
  5. Multimodal Fusion: The aligned features are then used for downstream tasks like sentiment analysis (predicting emotion) and relation detection (identifying how entities in a sentence are connected).
Figure 1: The architecture of our model, where EEG and language features are coordinately explored by two encoders. The EEG encoder and language encoder are shown on the left and right, respectively. The cross-alignment module is used to explore the connectivity and relationship within two domains, while the transformed features are used for downstream tasks.

Datasets

The authors evaluated their model on two primary datasets:

  • ZuCo (1.0 and 2.0): A corpus of EEG signals and eye-tracking data recorded while participants read natural English sentences (including movie reviews from the Stanford Sentiment Treebank and Wikipedia articles).
  • K-EmoCon: A multimodal dataset containing physiological signals (including EEG) and emotion annotations from naturalistic conversations.

Key Insights and Ideas

  • Frequency Matters: The alignment weights revealed that beta2 and gamma1 frequency bands are the most active during sentiment processing. This aligns with existing neuroscience literature suggesting these bands are involved in higher-order linguistic functions and emotion detection.
  • Spatial Specialization: Brain topographical maps showed that important positive words (like "upscale") trigger significant activity in Broca’s area (speech production) and Wernicke’s area (comprehension).
  • Negative Word Paradox: Interestingly, negative words (like "lame") showed lower activation in traditional language centers but higher activation in the occipital lobes (visual processing) and inferior parietal lobes (attention/memory). The authors suggest this might be because negative events are often more "sticky" in memory, requiring different cognitive resources.
  • Alignment Improves Clustering: Using t-SNE visualizations, the authors showed that their alignment module makes features much more separable and less sparse, which directly explains the performance boost in downstream tasks.
Figure 4: Brain topologies.

Key Results

MTAM significantly outperformed existing unimodal and multimodal baselines:

  • Sentiment Analysis: Achieved a 9.3% F1-score improvement on the ZuCo dataset and a 1.7% improvement on K-EmoCon.
  • Relation Detection: Achieved a 7.4% F1-score improvement on the ZuCo dataset.
  • Word-Level Superiority: The experiments showed that aligning EEG at the word level (matching specific brain responses to specific words) generally outperforms sentence-level alignment.

Limitations and Future Work

The authors identify three main hurdles for this line of research:

  1. Data Scarcity: Collecting high-quality EEG data paired with natural language is difficult, time-consuming, and raises privacy concerns. Current datasets are relatively small compared to standard NLP benchmarks.
  2. Model Complexity vs. Overfitting: Because the datasets are small, the authors had to keep the MTAM architecture relatively straightforward. Adding more layers often led to overfitting rather than better generalization.
  3. Lack of Baselines: As a pioneer in this specific type of computational alignment, there were few existing published works to compare against, requiring the authors to implement many of their own baselines (MLP, Bi-LSTM, etc.) from scratch.

Future work will likely focus on collecting larger-scale brain-language datasets and exploring more complex transformer architectures as more data becomes available.

Summary updated

Abstract

Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced \textbf{MTAM}, a \textbf{M}ultimodal \textbf{T}ransformer \textbf{A}lignment \textbf{M}odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at \url{https://github.com/Jason-Qiu/EEG_Language_Alignment}.
BibTeX
@inproceedings{han2023can,
  author = {William Han and Jielin Qiu and Jiacheng Zhu and Mengdi Xu and Douglas Weber and Bo Li and Ding Zhao},
  title = {{Can Brain Signals Reveal Inner Alignment with Human Languages?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=uxzlH5bLrJ},
}
Figure