Listening with Great Expectations: An Investigation of Word Form Anticipations in Naturalistic Speech

M. Bentum, L. ten Bosch, A. van den Bosch, Mirjam ErnestusInterspeech 2019

TL;DR

This study investigates how our brains predict upcoming words in natural speech using electroencephalography (EEG). By measuring the "phonological mismatch negativity" (PMN) brainwave, researchers found that even subtle mismatches between expected and actual word sounds incur a processing cost, suggesting our brains are constantly making predictions across the entire spectrum of word predictability, not just in extreme cases.

What Problem Does This Paper Address?

Understanding how humans process spoken language in real-time is a long-standing challenge. A key debate revolves around the interplay of "top-down" (predictive) and "bottom-up" (sensory-driven) processing. While some models suggest predictions only influence higher-level word recognition, others propose they can affect lower-level auditory processing. This paper aims to test the predictive coding framework's hypothesis that our brains generate predictions about incoming sensory information and that mismatches between these predictions and reality signal an error, driving further processing. Specifically, the authors want to know if these predictions are active even when words are highly predictable, or only in rare, surprising situations.

Methods

The core of this study lies in using the phonological mismatch negativity (PMN), an event-related potential (ERP) component in EEG signals. The PMN is thought to reflect the mismatch between expected and perceived auditory speech input.

The researchers developed a novel measure of this mismatch based on the predictive coding framework. This involves:

  1. Estimating Top-Down Expectations: A statistical language model (SLM) was used to predict the probability of a word given the preceding words. This represents the listener's "prior expectation."
  2. Representing Bottom-Up Input: Automatic speech decoding techniques were adapted to estimate the probability of phoneme sequences from a partial auditory fragment of the current word.
  3. Updating Expectations: These phoneme probabilities were used to update the prior word probabilities, creating a "post WPD" (word probability distribution) that incorporates the auditory input.
  4. Quantifying Mismatch: The cross-entropy between the prior and post WPDs was calculated for each word. A higher cross-entropy indicates a greater mismatch between the listener's expectation and the actual auditory input.

This cross-entropy measure was then correlated with the amplitude of the PMN recorded from participants listening to naturalistic speech.

Datasets

  • Speech Corpus: Approximately 50,000 word tokens extracted from two Dutch corpora: the Spoken Dutch Corpus [19] and IFADV [20]. This corpus was used for the EEG experiment and consisted of spontaneous dialogues, read-aloud stories, and news broadcasts.
  • NLCOW14 (COW): A large collection of web-crawled Dutch texts (4.7 billion words) used to train the statistical language model for estimating word probabilities.

The authors note that their Speech Corpus, while large, required significant artifact removal (~50% of word epochs), but still resulted in over a million word epochs for analysis, a much larger dataset than typically used in classical EEG experiments.

Key Insights and Ideas

  • Ubiquitous Prediction: The study provides strong evidence that listeners actively anticipate auditory input across the entire probability spectrum of word predictability, not just in cases of high surprise. This challenges the notion that prediction is only a mechanism for dealing with unexpected events.
  • Mismatch as a Processing Cost: A higher cross-entropy (greater mismatch between expectation and auditory input) consistently led to a more negative PMN amplitude. This suggests that even small deviations from expectations incur a processing cost, reflected in brain activity.
  • Top-Down Feedback is Crucial: The findings support the predictive coding framework, emphasizing the role of top-down feedback in perception. The brain's internal model generates predictions, and mismatches serve as error signals to update these models and guide processing.
  • Naturalistic Speech is Viable for EEG: The study successfully demonstrates the feasibility of analyzing EEG data from long stretches of naturalistic speech, overcoming some of the artifact challenges to gain more ecologically valid insights.

Key Results

  • A statistically significant negative correlation was found between the cross-entropy measure and the PMN amplitude.
  • Specifically, a higher cross-entropy value was associated with a more negative PMN amplitude, indicating that greater mismatches between expected and perceived word forms lead to stronger neural responses.
  • The model incorporating cross-entropy significantly improved the prediction of ERP amplitudes compared to a model without it (χ² = 4.86, p < .05).
  • The analysis included approximately 1.17 million word epochs from 48 participants, representing 93% of the words in the speech materials after artifact rejection.

Limitations and Future Work

  • Artifacts in Naturalistic Speech: While the study successfully used naturalistic speech, a significant portion of the data had to be removed due to artifacts (around 50% of word epochs). This highlights the ongoing challenge of artifact management in such datasets.
  • Random Effects Convergence: A random slope for cross-entropy by participant could not be included in the final statistical model due to convergence errors, suggesting potential variability in individual responses.
  • Future Directions: The authors suggest that their approach, which quantifies mismatch across the whole probability spectrum, provides a more nuanced understanding of predictive language processing than studies focusing only on extreme predictability. Future work could further explore the implications of these findings for models of speech perception and learning.

Summary updated

BibTeX
@inproceedings{bentum2019listening,
  author = {M. Bentum and L. ten Bosch and A. van den Bosch and Mirjam Ernestus},
  title = {{Listening with Great Expectations: An Investigation of Word Form Anticipations in Naturalistic Speech}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure