Aligning Brains into a Shared Space Improves their Alignment to Large Language Models

Arnab Bhattacharjee, Zaid Zada, Haocheng Wang, Bobbi Aubrey, Werner Doyle, Patricia Dugan, Daniel Friedman, Orrin Devinsky, Adeen Flinker, Peter J. Ramadge, Uri Hasson, Ariel Goldstein, Samuel A. NastasebioRxiv 2024 · DOI

TL;DR

This paper introduces a "shared response model" (SRM) to create a common neural representation space across individuals listening to the same audio. This shared space significantly improves the ability of large language models (LLMs) to predict brain activity and allows for more generalized encoding models that work across different people.

What Problem Does This Paper Address?

Predicting human brain activity during natural language processing using Large Language Models (LLMs) has shown great promise. However, existing methods typically build separate predictive models for each individual and each electrode. This approach doesn't tell us how well these models generalize across different people, which is a crucial step towards understanding shared neural representations of language and building more robust brain-computer interfaces. The challenge is that even when listening to the same stimulus, individual brains have unique "topographies" of activity due to differences in electrode placement and functional organization. This paper aims to overcome this by creating a unified neural space that captures commonalities across individuals.

Methods

The core of this work is the Shared Response Model (SRM), adapted from fMRI research for use with electrocorticography (ECoG) data. SRM learns subject-specific transformations to map individual neural activity into a common, shared feature space. This shared space is designed to capture stimulus-driven neural responses that are consistent across individuals, while filtering out idiosyncratic variations.

The process involves:

  1. Estimating a Shared Space: SRM is trained on a subset of the data from multiple subjects to learn a shared representation space and individual transformation matrices.
  2. LLM Embeddings: Contextual word embeddings are extracted from the GPT-2 XL language model for each word in the podcast. These embeddings are then reduced in dimensionality using PCA.
  3. Encoding Models: Linear encoding models are built to map these LLM embeddings to the neural activity (either in the shared space or original electrode space).
  4. Denoising and Reconstruction: The shared space is used to "denoise" individual brain signals by projecting them back into the electrode space. This reconstructed signal is then used for encoding.
  5. Cross-Subject Generalization: The study also tests how well encoding models trained in the shared space on a subset of subjects can predict the brain activity of a new, unseen subject.

Datasets

The study used electrocorticography (ECoG) data from 8 human participants who listened to a 30-minute audio podcast titled "So a Monkey and a Horse Walk Into a Bar, Act One: Monkey in the Middle” from This American Life. The podcast transcript was manually aligned with the audio. The LLM used was GPT-2 XL.

Key Insights and Ideas

  • Unifying Neural Signals: The SRM effectively creates a common ground for neural activity across individuals, even with varying electrode placements. This is crucial for understanding shared language processing.
  • LLMs as Neural Predictors: LLMs, particularly their contextual embeddings, are powerful tools for predicting brain activity during natural language comprehension.
  • Denoising through Shared Space: Projecting individual brain signals into the shared space and back out acts as a form of denoising, improving the signal-to-noise ratio for encoding.
  • Language Areas Benefit Most: The improvements in encoding performance are most pronounced in brain regions known to be specialized for language comprehension, such as the superior temporal gyrus (STG) and inferior frontal gyrus (IFG).
  • Generalization is Key: The shared space enables encoding models to generalize to new subjects, a significant step towards building universal brain decoding tools.

Key Results

  • Improved Encoding Performance: Using the SRM-based shared space significantly improved LLM-based encoding model performance compared to using individual electrodes directly.
  • Denoising Boost: Reconstructing individual brain signals via the shared space resulted in a mean 38% improvement in peak encoding performance across subjects.
  • Regional Specificity: The most substantial improvements were observed in the inferior frontal gyrus (IFG) and superior temporal gyrus (STG).
  • Cross-Subject Generalization: Encoding models trained in the shared space on a subset of subjects performed comparably to, and sometimes better than, within-subject models when tested on a new subject.
  • SRM vs. PCA: The SRM outperformed a Principal Component Analysis (PCA) control, demonstrating that the alignment of neural trajectories, not just dimensionality reduction, is key to the performance gains.

Limitations and Future Work

  • Limited Subject Number: The study was conducted on 8 subjects. While this is a reasonable number for ECoG studies, a larger cohort would further strengthen the findings on generalization.
  • Stimulus Specificity: The study used a single podcast. Future work could explore how well the SRM generalizes across different types of naturalistic stimuli (e.g., stories, conversations, lectures).
  • LLM Choice: While GPT-2 XL was used, exploring other LLMs and their architectures could reveal further insights into the relationship between language models and brain activity.
  • Clinical Applications: The authors suggest that this approach could accelerate research on individualized brain decoding and brain-computer interfaces, especially for individuals with limited data availability (e.g., epilepsy patients). This points towards future work in developing practical applications.

Summary updated

Abstract

A bstract Recent studies have shown that large language models (LLMs) can accurately predict neural activity measured using electrocorticography (ECoG) during natural language processing. To predict word-by-word neural activity, most prior work has estimated and evaluated encoding models within each electrode and subject—without evaluating how these models generalize across individual brains. In this paper, we analyze neural responses in 8 subjects while they listened to the same 30-minute podcast episode. We use a shared response model (SRM) to estimate a shared information space across subjects. We show that SRM significantly improves LLM-based encoding model performance. We also show that we can use this shared space to denoise the individual brain responses by projecting back into the individualized electrode space, and this process achieves a mean 38% improvement in encoding performance. The strongest improvement was observed for brain areas specialized for language comprehension, specifically in the superior temporal gyrus (STG) and inferior frontal gyrus (IFG). Critically, estimating a shared space allows us to construct encoding models that better generalize across individuals.
BibTeX
@misc{bhattacharjee2024aligning,
  author = {Arnab Bhattacharjee and Zaid Zada and Haocheng Wang and Bobbi Aubrey and Werner Doyle and Patricia Dugan and Daniel Friedman and Orrin Devinsky and Adeen Flinker and Peter J. Ramadge and Uri Hasson and Ariel Goldstein and Samuel A. Nastase},
  title = {{Aligning Brains into a Shared Space Improves their Alignment to Large Language Models}},
  year = {2024},
  doi = {10.1101/2024.06.04.597448},
  howpublished = {bioRxiv},
}
Figure