The intelligibility of lombard speech: communicative setting matters

Michael Fitzpatrick, Jeesun Kim, Chris DavisInterspeech 2012

TL;DR

This paper investigates how the visual context of a conversation (face-to-face vs. non-visual) affects the intelligibility of speech produced in noise (Lombard speech). They found that while auditory intelligibility didn't differ significantly, visual cues significantly boosted the intelligibility of Lombard speech when participants could see each other, suggesting that talkers adapt their speech production to leverage available communication channels.

What Problem Does This Paper Address?

When we speak in noisy environments, our speech naturally changes – it becomes louder, slower, and our articulation becomes more exaggerated. This phenomenon is known as the Lombard effect. Previous research has shown that talkers modify their speech not only acoustically but also visually when speaking in noise. However, the communicative setting – specifically, whether the interlocutor can be seen – has been hypothesized to influence these modifications. This paper aims to bridge this gap by examining whether these setting-dependent speech modifications (face-to-face vs. non-visual) translate into differences in speech intelligibility, both when listening to speech alone and when also seeing the speaker.

Methods

The researchers used speech recordings from a previous study where participants engaged in a game requiring them to repeat consonant-vowel-consonant (CVC) tokens in quiet and in noise, under both face-to-face (FTF) and non-visual (NV) conditions.

For the current study, they created auditory-only (AO), visual-only (VO), and auditory-visual (AV) stimuli from these recordings. The auditory stimuli were mixed with speech-shaped noise at a -10dB SNR. Participants were then asked to identify the CVC tokens presented under these different conditions.

Key to their analysis were:

  • Auditory-Only (AO) conditions: Assessing intelligibility based solely on sound.
  • Visual-Only (VO) conditions: Assessing intelligibility based solely on lip movements.
  • Auditory-Visual (AV) conditions: Assessing intelligibility with both auditory and visual information.
  • AV Relative Benefit (RB): A metric to quantify the additional intelligibility gained from visual cues, normalized by the auditory-only performance.

Statistical analysis involved 3-way repeated measures ANOVAs to examine the effects of modality (AO, VO, AV), production condition (NV, FTF), and speaking style (Quiet, Lombard).

Datasets

The study utilized speech recordings collected in a previous experiment ([1]). These recordings consisted of consonant-vowel-consonant (CVC) tokens (specifically "hAd, hARd, hEEd, hId, hod, hUd") produced in quiet and in noise, under both face-to-face (FTF) and non-visual (NV) communicative settings. The stimuli were carefully selected to avoid co-articulation and idiosyncratic speaking behaviors.

Key Insights and Ideas

  • Visual cues matter most in FTF Lombard speech: While auditory intelligibility of Lombard speech was similar regardless of whether the talker was seen or not, the intelligibility of speech when both auditory and visual information were available (AV) was significantly higher in the FTF condition compared to the NV condition.
  • Lombard speech benefits more from AV interaction: The relative benefit of visual speech (AV RB) was greater for Lombard speech than for quiet speech, indicating that the exaggerated visual cues in Lombard speech are particularly helpful when combined with auditory information.
  • FTF enhances AV benefit for Lombard speech: The AV relative benefit for Lombard speech was significantly higher in the FTF condition than in the NV condition. This suggests that when talkers can see their interlocutor, they produce visual speech cues that are more beneficial for AV intelligibility in noisy conditions.
  • Auditory-only Lombard speech is less sensitive to setting: Interestingly, there was no significant difference in the auditory-only intelligibility of Lombard speech between the FTF and NV conditions. This implies that the primary advantage of the FTF setting for Lombard speech intelligibility comes from the visual channel.

Key Results

  • Auditory Only (AO): No significant difference in intelligibility for Lombard speech between NV and FTF conditions.
  • Visual Only (VO): Lombard speech produced in the FTF condition was perceived significantly more accurately than NV Lombard speech. Quiet speech showed no significant difference.
  • Auditory Visual (AV):
    • For Quiet speech, NV items were perceived more accurately than FTF items.
  • For Lombard speech, FTF items were perceived significantly more accurately than NV items.
  • AV Relative Benefit (RB):
    • Averaged across NV and FTF, the benefit from visual speech was greater for Lombard speech than quiet speech.
  • For Lombard speech, the AV RB was significantly greater for FTF (74.39%) compared to NV (52.17%).
  • For quiet speech, the AV RB was higher for NV (55.91%) compared to FTF (43.52%).

Limitations and Future Work

  • Limited visual information: The study acknowledges that the visual stimuli were constrained to lip and mouth movements, potentially excluding other informative visual cues like head movements. Future work could explore the impact of more complete visual information.
  • Auditory normalization: The normalization of auditory amplitude might have reduced acoustic differences between NV and FTF Lombard speech, potentially masking some auditory intelligibility benefits. A follow-up study using original, non-normalized stimuli is planned.
  • Viseme confusability: The study used a limited set of CVC tokens, and not all vowels were equally confusable. Further research with a wider range of stimuli is needed to understand how viseme categories are affected by Lombard speech.
  • Focus on lip movements: The authors suggest extending their findings to more complete videos to examine which visual regions are most informative for AV Lombard speech perception.

Summary updated

BibTeX
@inproceedings{fitzpatrick2012intelligibility,
  author = {Michael Fitzpatrick and Jeesun Kim and Chris Davis},
  title = {{The intelligibility of lombard speech: communicative setting matters}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure