Abstract
Electroencephalography (EEG) and language have been widely explored independently for many downstream tasks (e.g., sentiment analysis, relation detection, etc.). Multimodal approaches that study both domains have not been well explored, even though in recent years, multimodal learning has been seen to be more powerful than its unimodal counterparts. In this study, we want to explore the relationship and dependency between EEG and language, i.e., how one domain reflects and represents the other. To study the relationship at the representation level, we introduced MTAM, a Multimodal Transformer Alignment Model, to observe coordinated representations between the two modalities, and thus employ the transformed representations for downstream applications. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure low-level language and EEG features to high-level transformed features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 16.5% on sentiment analysis for K-EmoCon, 27% on sentiment analysis of ZuCo, and 31.1% on relation detection of ZuCo. In addition, we provide interpretations of the performance improvement by: (1) visualizing the original feature distribution and the transformed feature distribution; (2) visualizing word-level and sentence-level EEG-language alignment weights; and (3) visualizing brain topographical maps to provide an intuitive demonstration of the connectivity of EEG and language response in the brain regions.
Introduction
While modern Large Language Models (LLMs) demonstrate remarkable capabilities, they remain computationally suboptimal compared to the human brain, which processes complex linguistic context with significantly less data and energy. Historically, brain signals (EEG) and natural language have been studied as independent modalities. While it is established that the brain reacts to linguistic stimuli, the computational dependency between these two domains remains under-explored. This research addresses a fundamental question: Can we mathematically align the neural encoding of a sentence with its computational representation? By mapping this alignment, we can potentially enhance machine performance on human-centric tasks like emotion recognition and relation detection.
Methodology
The researchers propose the Multimodal Transformer Alignment Model (MTAM), which utilizes a bi-encoder paradigm to bridge the gap between neural and linguistic data.
As shown in the architecture above, the model consists of three primary components:
- Hierarchical Transformer Encoders: Separate encoders process raw EEG signals and text (leveraging pre-trained models like BERT) to generate high-level feature vectors.
- Cross Alignment Module (CAM): This module transfigures disparate features into a shared representation space using two loss functions: Canonical Correlation Analysis (CCA) to maximize linear correlation, and Wasserstein Distance (WD) to align the underlying geometry of the distributions.
- Downstream Task Heads: The aligned multimodal features are passed to task-specific layers for final classification.
Experimental Setup
The study evaluates MTAM using two primary multimodal datasets:
- ZuCo (1.0 and 2.0): Contains EEG and eye-tracking data from subjects reading movie reviews (sentiment) and Wikipedia snippets (relation detection).
- K-EmoCon: A dataset focused on emotion recognition during naturalistic social conversations, pairing EEG signals with emotional labels.
Results and Discussion
MTAM achieved new state-of-the-art (SOTA) performance across all evaluated tasks. In sentiment analysis, the model saw an F1-score improvement of 9.3% on ZuCo and 1.7% on K-EmoCon. For relation detection on ZuCo, performance improved by 7.4%. Crucially, the multimodal approach (EEG + Text) consistently outperformed unimodal baselines, confirming that brain signals provide unique, complementary information that text alone cannot capture.
Interpretability analysis revealed several key insights:
- Frequency Band Significance: Alignment weights were highest in the Beta2 and Gamma1 bands. This corresponds with neuroscience findings linking Gamma waves to emotion detection and Beta waves to higher-order linguistic processing.
- Spatial Connectivity: Brain topographical maps (illustrated above) show that sentiment-rich words like "upscale" or "lame" trigger concentrated activity in Wernicke’s area (comprehension) and Broca’s area (production). Conversely, neutral or unimportant words shifted activity toward the occipital lobe, associated with basic visual processing.
- Feature Separability: t-SNE visualizations demonstrated that while raw EEG and text features are often overlapping, the MTAM alignment process produces tighter, more distinct clusters, facilitating easier classification.
Conclusion
This work demonstrates that aligning brain activity with language representations significantly enhances NLP task performance. While limited by the relatively small size of current EEG datasets, the results suggest a promising future for "brain-aligned" machine learning. Future research will focus on scaling data collection and exploring whether these human-centric representations can be distilled into LLMs to improve their understanding without requiring live EEG input during inference.
Summary updated