A Framework for Decoding Event-Related Potentials from Text

Shaorong Yan, Aaron Steven WhiteNAACL 2019

TL;DR

This paper introduces a novel framework for decoding brain activity (ERPs) during reading by combining pre-trained convolutional neural networks with language models. They demonstrate that while modern contextual word embeddings alone underperform simpler surprisal-based models, combining them leads to the best ERP reconstruction, suggesting a nuanced interplay between different linguistic processing mechanisms.

What Problem Does This Paper Address?

Understanding how the human brain processes language in real-time is a central goal in cognitive science and psycholinguistics. Event-Related Potentials (ERPs), a measure of brain activity time-locked to specific events (like word presentation), offer excellent temporal resolution for studying these rapid cognitive processes. However, traditional ERP research often relies on analyzing specific components (e.g., N400, P600), which can be problematic. Defining these components can be subjective, and their fixed time windows might miss transient effects or conflate the influence of different linguistic factors. This can lead to researcher degrees of freedom in analysis and complex statistical challenges. The paper aims to address this by proposing a more direct, data-driven approach to decoding ERPs from linguistic input, reducing analytical subjectivity and allowing for a more direct comparison of different computational models of language processing.

Methods

The core of the paper is a novel framework that uses a convolutional neural network (CNN) decoder pre-trained as part of an autoencoder. This decoder is designed to reconstruct ERP data from a compressed representation. The key innovation is decoupling this decoder and coupling it with various language models to extract features.

Here's a breakdown:

  • Convolutional Decoder: A CNN architecture (specifically, a transposed convolutional network) is trained to reconstruct ERP data from a compressed representation. This decoder is pre-trained using an autoencoder setup, where a convolutional encoder compresses the ERPs and the decoder reconstructs them. The best-performing decoder architecture (chosen via cross-validation) is then frozen.
  • Language Model Integration: Instead of the original autoencoder's encoder, various language models are used to extract features for each word in a sentence. These features are then mapped to the decoder's expected input space via a learned "interface mapping" (a linear transformation).
  • Language Models Explored:
    • Surprisal: Lexical surprisal (how predictable a word is given its preceding context), derived from a Recurrent Neural Network (RNN).
  • Semantic Distance: Cosine distance between a word's embedding and the average embedding of its preceding context (using GloVe embeddings).
  • Static Word Embeddings: Direct use of GloVe embedding dimensions.
  • Contextual Word Embeddings: Embeddings from ELMo, which capture word meaning in context.
  • Combined Models: Various combinations of the above features.

This framework allows for a direct comparison of how well different linguistic theories (represented by features from these language models) can predict neural activity.

Datasets

The study uses a dataset collected by Frank and Willems (2017). This dataset consists of ERPs from 24 subjects who read sentences from natural text presented word-by-word. The ERPs are epoched from -100 to 700ms relative to word onset, from 32 scalp channels. After artifact rejection, the dataset contains 41,009 training instances. No new benchmarks or datasets were introduced.

Key Insights and Ideas

  • Surprisal Still Reigns (Partially): Simpler, surprisal-based models surprisingly outperform more complex, modern contextual word embeddings (like ELMo) when used alone for ERP prediction. This suggests that the fundamental predictability of a word in context is a very strong signal for neural processing.
  • The Power of Combination: The best performance is achieved not by any single model, but by combining surprisal with contextual word embeddings (ELMo). This highlights that different linguistic features capture distinct aspects of neural processing, and a comprehensive model needs to integrate them.
  • Contextual Embeddings Add Value: While not outperforming surprisal alone, contextual embeddings (ELMo) do add significant predictive power when combined with surprisal and semantic distance, suggesting they capture nuances beyond simple prediction or static semantic similarity.
  • Time-Course Matters: The analysis of how different predictors contribute over time reveals that frequency, surprisal, and semantic distance influence ERPs in distinct temporal windows, broadly aligning with known ERP components like N1, N250, and N400, but offering a more continuous view.
  • Beyond Perceptual Features: The success of models that don't explicitly encode orthographic or phonetic information (except for ELMo's character CNN) in capturing early ERP effects suggests that top-down linguistic processing plays a significant role even in early stages of word recognition.

Key Results

  • The beta model (a CNN architecture) without subject-specific intercepts was selected as the best decoder.
  • The combined model of frequency, surprisal, and semantic distance plus ELMo embeddings achieved the highest performance, explaining 49.5% of the variance (R²mod) with a 95% confidence interval of [48.9, 50.1].
  • Models based on frequency alone explained only 19.5% of the variance.
  • Models combining surprisal and semantic distance (F+S+SD) explained 46.6% of the variance.
  • Adding ELMo embeddings to the surprisal and semantic distance model (F+S+SD+ELMO) improved performance from 46.6% to 49.5%.
  • Part-of-speech analysis showed models generally performed better for content words than function words.

Limitations and Future Work

  • Limited to Reading: The framework is currently applied to ERPs collected during reading. Extending it to other linguistic tasks (e.g., speech comprehension) would be valuable.
  • Decoder Architecture: While CNNs are powerful, exploring other neural architectures for the decoder could yield further improvements.
  • Language Model Choices: The study focused on specific language models. Future work could explore newer or different types of language models.
  • Probing NLP Models: The authors suggest that this framework is not only useful for comparing cognitive models but also for "probing and improving the representations constructed by natural language processing (NLP) systems," especially for understanding complex neural models. This opens a significant avenue for future research.
  • Time-Course Nuances: While the time-course analysis showed distinct contributions, further detailed investigation into the precise temporal dynamics and interactions between different linguistic features is warranted.

Summary updated

BibTeX
@inproceedings{yan2019framework,
  author = {Shaorong Yan and Aaron Steven White},
  title = {{A Framework for Decoding Event-Related Potentials from Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure