Effect of being seen on the production of visible speech cues. a pilot study on lombard speech

Maëva Garnier, Lucie Ménard, Gabrielle RichardInterspeech 2012

TL;DR

This study investigates whether speakers unconsciously enhance their visible speech cues (like lip movements) when speaking in noise, especially if their communication partner can see them. The findings suggest that while speakers do modify their speech in noise (the Lombard effect), this enhancement isn't specifically targeted at improving visual intelligibility by amplifying lip movements. Instead, the observed changes seem to be a more general correlate of increased vocal effort.

What Problem Does This Paper Address?

We know that seeing someone speak significantly improves our ability to understand them, especially in noisy environments. This is known as audiovisual intelligibility. Simultaneously, when people speak in noise, they exhibit the "Lombard effect," characterized by increased vocal effort, higher pitch, and importantly, amplified speech articulation, particularly lip movements. This raises a crucial question: Is this hyper-articulation a deliberate, albeit unconscious, strategy by speakers to boost their own visual intelligibility when they know they're being seen? Or is it simply a byproduct of speaking louder? This paper aims to disentangle these possibilities by examining how different interaction modalities (audio-only vs. audio-visual) affect speech production in noise.

Methods

The researchers used a single French Canadian speaker who performed a speech task in both quiet and a noisy (85 dB) environment. They employed three interaction conditions:

  • No Interaction (NI): The speaker read sentences aloud without any direct interaction.
  • Audio Only (AO): The speaker gave instructions to an experimenter who could only hear them.
  • Audio Visual (AV): The speaker interacted with an experimenter who could both see and hear them.

To capture speech production, they used:

  • 3D Electromagnetic Articulography: This is the core novel technique here, allowing for precise, synchronous tracking of lip, jaw, and tongue movements at 200 Hz. This is a significant advancement over previous methods that might have focused on fewer articulators or less precise tracking.
  • Microphone: To record the audio signal, which was later processed to remove noise and analyze acoustic features like intensity and formant frequencies (F1, F2) using Praat.

The study focused on specific target words embedded in carrier sentences, repeated multiple times across conditions to ensure robust data.

Datasets

Not discussed in the paper. The study used a single speaker and a controlled experimental setup.

Key Insights and Ideas

  • Lombard Effect is Real, But Not Necessarily Strategic for Vision: The study confirms that speaking in noise leads to increased vocal intensity and altered acoustic properties of vowels. However, the key insight is that the amplification of visible lip movements wasn't significantly greater when the speaker was being seen (AV condition) compared to when they were only being heard (AO condition).
  • Less Visible Articulators Aren't "Compensating": Tongue movements, which are less visible, did not show a specific enhancement in noise, regardless of the interaction modality. Their movement seemed to follow the general increase in jaw aperture. This suggests that the speaker isn't strategically enhancing less visible cues to compensate for potential visual degradation.
  • Audio-Only Interaction Amplifies Lip Movements More: Surprisingly, the most significant amplification of lip movements occurred in the audio-only condition, not the audio-visual one. This suggests that the speaker might be trying to make their speech more acoustically salient in a purely auditory context, rather than visually.
  • Visible Contrast is Enhanced, But Indirectly: While the visible contrast between different vowels (e.g., lip aperture for /a/ vs. /ε/) was enhanced in the AO condition, this appears to be a consequence of the overall increased articulation, not a targeted visual strategy.

Key Results

  • Vocal Intensity: Increased in noise across all conditions, with a greater increase in interactive (AO, AV) vs. non-interactive (NI) conditions. The increase was also greater in AO than AV.
  • Formant Frequencies (F1, F2): Modifications were observed, with some contrasts (like vowel height) being better preserved in AO than NI or AV.
  • Lip Articulation:
    • In the NI condition, lip movements were not amplified in noise compared to silence.
  • A slight increase in lip aperture was seen in the AV condition.
  • The greatest modification of lip articulation (increased aperture and spreading, decreased protrusion) occurred in the AO condition.
  • Tongue Articulation: Weak changes (<1mm) were observed in NI. In AV, tongue position didn't change much, but jaw lowered. In AO, a greater displacement of the tongue downwards and forward was observed for all vowels.

Limitations and Future Work

  • Single Speaker: The most significant limitation is that these findings are based on a single speaker. The authors explicitly state that further investigation with more speakers is needed to determine if these results are generalizable.
  • Pilot Study: The paper is explicitly labeled as a "pilot study," underscoring the need for broader research.
  • Future Work: The authors plan to investigate additional speakers to confirm their findings and explore whether the observed patterns hold across a wider population. They also aim to determine if the "expanding sonority" (increasing vocal intensity) is the primary strategy speakers employ to maintain intelligibility in noise, rather than enhancing visual cues.

Summary updated

BibTeX
@inproceedings{garnier2012effect,
  author = {Maëva Garnier and Lucie Ménard and Gabrielle Richard},
  title = {{Effect of being seen on the production of visible speech cues. a pilot study on lombard speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure