Talker Intelligibility and Listening Effort with Temporally Modified Speech

Maximillian Paulus, Valerie Hazan, Patti AdankInterspeech 2019

TL;DR

This paper investigates how modifying speech's temporal characteristics (like speaking rate) affects both how well people understand it and how much mental effort they exert. They found that while speeding up speech hurts intelligibility, slowing it down doesn't help, and surprisingly, both speeding up and slowing down speech increase listening effort, as measured by pupil dilation.

What Problem Does This Paper Address?

Understanding speech in noisy or challenging environments is a major hurdle, and researchers have explored various ways to improve it. While spectral enhancements (like filtering) are known to boost intelligibility, temporal modifications (like changing speaking rate) have had less consistent success. A crucial, yet often overlooked, aspect of speech perception is "listening effort" – the mental exertion required to understand speech. This is increasingly important for applications like assistive listening devices or voice assistants. Previous work has shown that pupil dilation is a reliable physiological measure of listening effort, increasing when intelligibility decreases. However, it's unclear how temporal speech modifications, independent of intelligibility changes, influence this effort. This paper aims to bridge this gap by examining the combined effects of talker characteristics, temporal modifications, and listening conditions on both intelligibility and listening effort.

Methods

The researchers conducted two experiments.

Experiment 1: * Participants: 16 normal-hearing native English speakers. * Speech Material: IEEE sentences spoken by two male talkers with naturally different speaking rates and intelligibility levels (T08: faster, more intelligible; T13: slower, less intelligible). * Temporal Modifications: * Plain: Unmodified speech. * Align: Locally aligned speech (using Vocalign) to synchronize temporal features. * Fast: Speaking rate increased by a factor of 2 (50% duration decrease). * Slow: Speaking rate decreased by a factor of 0.66 (52% duration increase). * Listening Conditions: * SMN (Speech-Modulated Noise): Noise shaped by the amplitude envelope of speech. * Nx (Noise Vocoder): Speech processed through a 10-channel noise vocoder. * Procedure: Participants listened to sentences in either SMN or Nx and performed keyword recall (for intelligibility) and had their pupil dilation measured (for listening effort). * Analysis: Linear mixed-effects models were used to analyze intelligibility (proportion of keywords recalled) and peak pupil dilation.

Experiment 2: * Participants: 24 normal-hearing native English speakers. * Speech Material: Similar to Experiment 1, but focused on a subset of modifications (Plain, Align). * Listening Condition: Only SMN was used. * Adaptive Procedure: Speech reception thresholds (SRTs) at 50% intelligibility were determined adaptively for each talker and modification. This allowed for a fixed intelligibility level across conditions, isolating the effect of temporal modification on listening effort. * Analysis: Similar to Experiment 1, but focused on peak pupil dilation at fixed intelligibility.

Pupil Data Preprocessing: Standard procedures were used, including gap filling for blinks, interpolation, outlier removal, and calculating percentage change from baseline. Peak dilation was defined as the maximum dilation within a specific time window after sentence onset.

Datasets

  • IEEE Sentences: A standard corpus of sentences used for speech intelligibility testing.
  • British English Language Model (BNC corpus): Used for calculating sentence entropy.
  • LTAS (Long-Term Average Spectrum): Used to generate speech-shaped noise for the SMN masker.
  • Non-experimental IEEE sentences: Used to create the modulation for the SMN masker.

No new datasets were introduced; existing standard datasets were utilized.

Key Insights and Ideas

  • Talker Intelligibility is Robust: The study confirms that a talker's inherent intelligibility (e.g., T08 being more intelligible than T13) is generally preserved across different listening conditions and modifications, aligning with previous research.
  • Temporal Modifications Have Complex Effects:
    • Intelligibility: Artificially increasing speaking rate (Fast) consistently reduced intelligibility. However, decreasing speaking rate (Slow or Align) did not improve intelligibility, contrary to the researchers' initial hypothesis. This suggests that simply slowing down speech isn't enough to boost understanding, especially in the presence of noise.
  • Listening Effort: This is where the most surprising findings emerged. While intelligibility didn't always change predictably with slower speech, pupil dilation (a measure of listening effort) increased for both artificially sped-up (Fast) and slowed-down (Align) speech, even when intelligibility was preserved. This suggests that temporal distortions, regardless of their direction, impose a cognitive load.
  • Pupil Dilation is Sensitive to Distortion: The study highlights that pupil dilation can capture subtle effects of signal processing that keyword recall might miss. For instance, the vocoded condition (Nx) in Experiment 1 showed increased pupil dilation, suggesting the spectral distortion itself contributed to listening effort, even if intelligibility was maintained.
  • Fixed Intelligibility Reveals Effort Differences: Experiment 2, by controlling for intelligibility, provided clearer evidence that temporal modifications (Align) can increase listening effort independently of how well the speech is understood.

Key Results

  • Experiment 1 (Intelligibility):
    • Talker T08 was significantly more intelligible than T13 (p < .001).
  • Fast speech was less intelligible than other modifications (p < .001).
  • Aligned speech was less intelligible than plain speech for T08 (p < .001), but not for T13.
  • Experiment 1 (Pupil Dilation):
    • While not statistically significant across all conditions due to sample size, descriptive trends showed increased pupil dilation for Fast speech and reduced dilation for T13's Aligned and Slow speech compared to Plain.
  • Nx condition showed larger peak pupil dilation than SMN.
  • Experiment 2 (Pupil Dilation):
    • Aligned speech was associated with larger peak pupil dilation for both talkers, even at fixed intelligibility levels (p < .05).

Limitations and Future Work

  • Sample Size: The authors acknowledge that the relatively small number of trials and participants in Experiment 1 might have limited the statistical power to detect significant effects for pupil dilation.
  • Temporal Modification Range: The study explored a limited range of temporal modifications. Future work should parametrically assess the effects of different degrees of temporal distortion.
  • Masker Types: While SMN was chosen to avoid informational masking, future studies could explore other masker types.
  • Auditory Models: The authors suggest that the observed effects might be related to disruptions in neural entrainment, a concept from neural models of speech processing. Further research could investigate these neural mechanisms.
  • Practical Applications: The findings could inform the design of audiological tests and software applications that allow users to control speech rate, potentially optimizing for both intelligibility and listening comfort.

Summary updated

BibTeX
@inproceedings{paulus2019talker,
  author = {Maximillian Paulus and Valerie Hazan and Patti Adank},
  title = {{Talker Intelligibility and Listening Effort with Temporally Modified Speech}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure