Information-making processes in the speaker’s brain drive human conversations

Ariel Goldstein, Haocheng Wang, Tom Sheffer, Mariano Schain, Zaid Zada, Leonard Niekerken, Bobbi Aubrey, Samuel A. Nastase, Harshvardhan Gazula, Colton Costo, Werner Doyle, Daniel Friedman, Sasha Devore, Patricia Dugan, Avinatan Hassidim, Michael P. Brenner, Yossi Matias, Orrin Devinsky, Adeen Flinker, Uri HassonbioRxiv 2024 · DOI

TL;DR

This study reveals that speakers' brains actively generate information-rich words through extra neural computation before speaking them, contrasting with listeners' brains which process such words after hearing them. This "information-making" process, distinct from information-seeking, requires more neural resources and time, suggesting our brains are active creators of novel linguistic content, not just passive transmitters.

What Problem Does This Paper Address?

Human conversation is a dynamic exchange of information. While we understand that listeners actively predict upcoming words and are surprised by unexpected ones (information-seeking), the process by which speakers generate these surprising, information-rich words on the fly remains largely a mystery. Information theory defines information as reducing uncertainty, but it doesn't fully explain the active cognitive effort involved in creating that uncertainty for the listener. This paper aims to bridge this gap by investigating the neural mechanisms behind "information-making" in the speaker's brain during spontaneous conversation.

Methods

The researchers employed a multi-faceted approach combining:

  • Information Theory & Large Language Models (LLMs): They used LLMs (Llama-2 and GPT-2) to quantify the "predictability" or "information content" of each word in a conversation. Words with low predicted probability were deemed "improbable" and "information-rich," while those with high probability were "probable" and "information-thin."
  • Electrocorticography (ECoG): They utilized high-density ECoG recordings from patients, capturing neural activity with excellent temporal and spatial resolution during natural conversations.
  • Neural Encoding Models: They built models to predict neural activity based on word embeddings extracted from LLMs. This allowed them to compare how the brain encodes probable versus improbable words.
  • Behavioral Analysis: They measured speech rate and pauses to identify behavioral markers associated with producing different types of words.

Datasets

The study used a unique dataset of approximately 100 hours of continuous, 24/7 spatiotemporal ECoG recordings from four patients engaged in spontaneous, real-life conversations. This dataset comprised roughly half a million words. The LLMs (Llama-2 and GPT-2) were used to analyze the predictability of these spoken words within their conversational context.

Key Insights and Ideas

  • Asymmetric Brain Processes: Listeners' brains prepare for predictable words before they are spoken, while speakers' brains show enhanced activity for improbable words before they are spoken. This highlights a fundamental difference between information-seeking and information-making.
  • "Information-Making" is Active: Generating surprising, information-rich words isn't effortless. It involves dedicated neural processes that recruit extra resources and time before articulation.
  • Beyond Word Frequency: The observed effects are not solely driven by how common a word is. The context in which a word appears significantly influences its predictability and the neural response, suggesting a deeper semantic and contextual processing.
  • LLMs Mirror Brains (Partially): LLMs also show increased internal computation when generating improbable words, mirroring the brain's effort. This suggests LLMs might be capturing some aspects of human language generation, but likely not the full picture.
  • Challenging Listener Expectations: The findings suggest speakers' brains actively aim to produce output that deviates from listener expectations, driving engagement and novelty in conversation.

Key Results

  • Behavioral Slowdown: Speakers took 100-150 ms longer to start articulating improbable words, indicating a behavioral marker for increased cognitive effort. This pause was independent of word frequency.
  • Pre-Articulation Neural Enhancement: In speakers, enhanced neural encoding was observed for improbable words 100-500 ms before they were spoken, particularly in language-related areas. This was widespread, unlike the more localized pre-onset activity for probable words in listeners.
  • Comprehension Contrast: In contrast, listeners showed enhanced encoding for probable words before word onset and for improbable words after word onset, consistent with prediction and surprise processing.
  • Robustness: The findings were robust across different LLMs (Llama-2, GPT-2), when analyzing only content words, and when using a shared set of words that were predictable in one context but not another.

Limitations and Future Work

  • Defining the "Neural Policy": The exact neural mechanisms or "policies" guiding the speaker's selection of informative words are still not fully defined.
  • LLM Limitations: While LLMs show similar patterns, their assessment of "surprise" is based on statistical probability and may not fully capture the nuances of shared human knowledge or context, potentially overestimating surprise.
  • Future Directions: The authors suggest further research into the neural policies for information-making, which is crucial for understanding human creativity, innovation, and the evolution of language and culture. They also highlight the need to move beyond entropy-based measures to fully grasp speech production.

Summary updated

Abstract

Abstract Natural human conversation is driven by the exchange of information-rich messages that surprise the listener and deviate from predictable context. While extensive research has characterized how the brain processes unexpected linguistic input during comprehension, the neural mechanisms underlying the generation of such information by the speaker remain poorly understood. In this study, we hypothesize that the speaker’s brain actively creates information through extra neural computation, transforming internal thoughts into novel, meaningful linguistic outputs. Utilizing a unique 24/7 dataset of continuous electrocorticography (ECoG) comprising approximately 100 hours of spontaneous natural conversations, we contrast the neural basis of speech production and comprehension within the same participants. Behaviorally, we find that speakers take longer pausing for an additional 100–150 milliseconds before producing information-rich, improbable words, even when controlling for word identity and frequency. Neurally, we provide converging evidence for a previously unreported process: generating information-rich words elicits significantly stronger neural activity (ERP) and enhanced neural encoding and decoding in language-related areas starting 100–500 ms before articulation. This pattern contrasts sharply with speech comprehension, where enhanced neural activity for predictable words occurs before onset, while responses to improbable words emerge only after onset as prediction errors. Furthermore, we demonstrate that large language models (LLMs) mirror this biological process, requiring deeper internal computation across layers to generate improbable versus probable words. Together, these results reveal that the speaker’s brain is not merely a transmission channel but an active generator of information, recruiting extra neural resources to create novel content that diverges from listener expectations.
BibTeX
@misc{goldstein2024information,
  author = {Ariel Goldstein and Haocheng Wang and Tom Sheffer and Mariano Schain and Zaid Zada and Leonard Niekerken and Bobbi Aubrey and Samuel A. Nastase and Harshvardhan Gazula and Colton Costo and Werner Doyle and Daniel Friedman and Sasha Devore and Patricia Dugan and Avinatan Hassidim and Michael P. Brenner and Yossi Matias and Orrin Devinsky and Adeen Flinker and Uri Hasson and Uri Hasson},
  title = {{Information-making processes in the speaker’s brain drive human conversations}},
  year = {2024},
  doi = {10.1101/2024.08.27.609946},
  howpublished = {bioRxiv},
}
Figure