Deep speech-to-text models capture the neural basis of spontaneous speech in everyday conversations

Ariel Goldstein, Haocheng Wang, Leonard Niekerken, Zaid Zada, Bobbi Aubrey, Tom Sheffer, Samuel A. Nastase, Harshvardhan Gazula, Mariano Schain, Aditi Singh, Aditi Rao, Gina Choe, Catherine Kim, Werner Doyle, Daniel Friedman, Sasha Devore, Patricia Dugan, Avinatan Hassidim, Michael P. Brenner, Yossi Matias, Orrin Devinsky, Adeen Flinker, Uri HassonbioRxiv 2023 · DOI

TL;DR

This paper uses a cutting-edge speech-to-text model (Whisper) to decode neural activity during natural, unconstrained conversations. They demonstrate that this model can predict brain responses to both the acoustic and semantic aspects of speech, revealing a hierarchical processing of language in the brain and a fascinating temporal interplay between speech production and comprehension.

What Problem Does This Paper Address?

Understanding how the human brain processes language in real-world, spontaneous conversations is a major challenge. Traditional neuroscience studies often rely on controlled, artificial environments that don't fully capture the complexity and dynamism of natural communication. This gap makes it difficult to build comprehensive models of language processing. The authors aim to bridge this gap by leveraging advanced deep learning models and a unique dataset of natural conversations to investigate the neural underpinnings of speech production and comprehension in their most natural form.

Methods

The core of this research lies in using a multimodal speech-to-text model called Whisper. Whisper is designed to process continuous speech and extract two types of "embeddings":

  • Speech embeddings: These capture acoustic features of speech.
  • Language embeddings: These capture the contextual structure and meaning of language.

The researchers then built electrode-wise encoding models. These models learn a linear mapping from Whisper's embeddings (both speech and language) to the neural activity recorded from individual electrodes in the brain. They used electrocorticography (ECoG) data from patients, which offers high spatiotemporal resolution. The models were trained on a large portion of the data and then tested on unseen conversations to assess their predictive power. They also employed t-SNE for visualizing the information encoded in the embeddings and performed statistical analyses to identify significant brain regions and temporal dynamics.

Datasets

The study utilized a novel, large-scale dataset of 100 hours of spontaneous, open-ended conversations from four patients. This dataset comprises approximately half a million words and includes concurrent ECoG recordings. The conversations were natural and unconstrained, involving interactions with family, friends, and hospital staff. This dataset is unique in its scale and naturalistic setting compared to typical controlled experimental paradigms.

Key Insights and Ideas

  • Deep learning models as windows into the brain: The success of Whisper's embeddings in predicting neural activity suggests that these models are capturing something fundamental about how the brain processes language.
  • Hierarchical language processing: The findings support a known hierarchy where sensory and motor areas are more sensitive to acoustic speech features, while higher-level language areas handle syntax and semantics.
  • Mixed selectivity is common: Many brain regions show a blend of sensitivity to both acoustic and linguistic information, suggesting a complex, integrated processing system.
  • Temporal flow of information: The study reveals a dynamic temporal sequence: during speech production, language information seems to precede acoustic information, while during comprehension, the reverse is observed. This highlights the predictive nature of the brain.
  • Real-world conversations are key: The ability to model neural activity in such unconstrained settings underscores the power of this approach for understanding natural human communication.

Key Results

  • High prediction accuracy: Encoding models achieved high accuracy in predicting neural responses to hundreds of thousands of words across many hours of left-out recordings. Correlations between predicted and actual neural signals reached up to 0.5.
  • Distinct roles for speech and language embeddings: Speech embeddings better predicted activity in sensory and motor areas (e.g., superior temporal gyrus, precentral gyrus), while language embeddings were more aligned with higher-order language areas (e.g., inferior frontal gyrus, angular gyrus).
  • Temporal dynamics:
    • During speech production, language encoding in the inferior frontal gyrus (IFG) peaked around 500 ms before word onset, while speech encoding in somatomotor areas peaked closer to word onset (~305 ms before).
  • During speech comprehension, speech encoding in the superior temporal gyrus (STG) peaked shortly after word onset (~55 ms), while language encoding in IFG peaked significantly later (~300 ms after).
  • Production vs. Comprehension differences: Different brain regions showed distinct preferences for encoding speech or language information depending on whether the patient was producing or comprehending speech.

Limitations and Future Work

  • Limited number of participants: The study involved only four patients, which limits the generalizability of findings.
  • ECoG data specific to epilepsy patients: While ECoG provides excellent temporal and spatial resolution, it is typically collected from individuals with epilepsy, which might introduce specific characteristics.
  • Future work: The authors suggest that this multimodal approach can be extended to predict neural responses in language areas for most conversations without needing to simplify or control speech. They also highlight the potential for deep learning models to serve as neurally-inspired cognitive models for language processing. Further research could explore the shared computational principles between deep learning models and the human brain in more detail.

Summary updated

Abstract

Abstract Humans effortlessly use the continuous acoustics of speech to communicate rich linguistic meaning during everyday conversations. In this study, we leverage 100 hours (half a million words) of spontaneous open-ended conversations and concurrent high-quality neural activity recorded using electrocorticography (ECoG) to decipher the neural basis of real-world speech production and comprehension. Employing a deep multimodal speech-to-text model named Whisper, we develop encoding models capable of accurately predicting neural responses to both acoustic and semantic aspects of speech. Our encoding models achieved high accuracy in predicting neural responses in hundreds of thousands of words across many hours of left-out recordings. We uncover a distributed cortical hierarchy for speech and language processing, with sensory and motor regions encoding acoustic features of speech and higher-level language areas encoding syntactic and semantic information. Many electrodes—including those in both perceptual and motor areas—display mixed selectivity for both speech and linguistic features. Notably, our encoding model reveals a temporal progression from language-to-speech encoding before word onset during speech production and from speech-to-language encoding following word articulation during speech comprehension. This study offers a comprehensive account of the unfolding neural responses during fully natural, unbounded daily conversations. By leveraging a multimodal deep speech recognition model, we highlight the power of deep learning for unraveling the neural mechanisms of language processing in real-world contexts.
BibTeX
@misc{goldstein2023deep,
  author = {Ariel Goldstein and Haocheng Wang and Leonard Niekerken and Zaid Zada and Bobbi Aubrey and Tom Sheffer and Samuel A. Nastase and Harshvardhan Gazula and Mariano Schain and Aditi Singh and Aditi Rao and Gina Choe and Catherine Kim and Werner Doyle and Daniel Friedman and Sasha Devore and Patricia Dugan and Avinatan Hassidim and Michael P. Brenner and Yossi Matias and Orrin Devinsky and Adeen Flinker and Uri Hasson},
  title = {{Deep speech-to-text models capture the neural basis of spontaneous speech in everyday conversations}},
  year = {2023},
  doi = {10.1101/2023.06.26.546557},
  howpublished = {bioRxiv},
}
Figure