Prosody for the eyes: quantifying visual prosody using guided principal component analysis

Erin Cvejic, Jeesun Kim, Chris Davis, Guillaume GibertInterspeech 2010

TL;DR

This paper quantifies visual cues of speech prosody (emphasis and phrasing) by analyzing facial and head movements. They introduce a novel application of guided Principal Component Analysis (gPCA) to extract interpretable visual features, revealing that both articulatory and non-articulatory movements (like eyebrow raises and head tilts) significantly differ based on prosodic intent. This work bridges the gap in understanding how visual information contributes to prosody perception.

What Problem Does This Paper Address?

Speech prosody, typically studied through auditory cues like pitch, duration, and intensity, also has a visual component. This visual prosody is conveyed through subtle movements of the face and head, including articulators (lips, jaw) and non-articulatory features (eyebrows, head orientation). While acoustic prosody is well-understood, the visual correlates are less clear due to challenges in consistent measurement and generalization across speakers and utterances. Previous studies often used small datasets, focused on limited facial regions, or examined single stressed syllables, hindering a comprehensive understanding of how visual cues signal prosodic information like focus and phrasing. This paper aims to address this gap by systematically quantifying these visual cues across a larger speech corpus and a wider range of facial and head movements.

Methods

The core methodological innovation lies in the application of guided Principal Component Analysis (gPCA) to analyze the complex 3D motion capture data of facial and head movements. Standard PCA finds orthogonal components that maximize variance, but gPCA is guided by prior knowledge to extract components that are interpretable in terms of articulatory parameters (e.g., jaw opening, lip spreading). This allows researchers to link specific visual movements to known speech production mechanisms.

The process involved:

  1. Motion Capture: 38 markers were placed on speakers' faces and heads, capturing 3D movements at 60Hz using an Optotrak 3020 system.
  2. Data Preprocessing: Rigid head movements were separated from non-rigid facial movements. Rigid movements were analyzed using standard PCA, while non-rigid movements were analyzed using gPCA.
  3. gPCA Application: The gPCA was guided by pre-defined articulatory parameters (listed in Table 1), resulting in 14 interpretable principal components (PCs) per frame.
  4. Feature Extraction: The "area under the curve" of these PCs over time was calculated for critical words within utterances, serving as a measure of the strength of each visual feature.
  5. Statistical Analysis: Paired samples t-tests with Bonferroni correction were used to compare these visual features across different prosodic conditions (broad focus, narrow focus, echoic question).

Datasets

The study utilized a custom dataset created for this research.

  • Speech Corpus: 30 non-expressive sentences from the IEEE Harvard Sentence list were used.
  • Prosodic Conditions: Each sentence was elicited in three prosodic conditions: broad focused statement, narrow focused statement, and echoic question.
  • Participants: Four native Australian English speakers participated.
  • Recordings: Each sentence in each condition was recorded twice, resulting in 180 sentences in total (30 sentences x 3 conditions x 2 tokens) per speaker.
  • Data Modalities: Synchronous audio, video, and 3D motion capture data were collected.

This dataset differs from previous work by its larger size and the inclusion of interactive dialog tasks to elicit naturalistic prosodic variations.

Key Insights and Ideas

  • Visual Prosody is Multifaceted: The study confirms that visual prosody isn't limited to just lip and jaw movements. Eyebrow movements and head rotations (pitch, roll, yaw) also play a significant role in signaling prosodic intent.
  • gPCA Unlocks Interpretability: Applying gPCA was crucial for extracting meaningful visual features. Instead of abstract components, they derived parameters directly related to speech production (jaw opening, lip spreading) and head motion, making the results more scientifically grounded.
  • Prosodic Differences Manifest Visually: Speakers exhibited distinct visual patterns for different prosodic conditions. Specifically, narrow focus and echoic questions elicited greater jaw and lip opening, and jaw protrusion compared to broad focus statements.
  • Non-Articulatory Cues are Important: Perhaps most surprisingly, non-articulatory movements like brow raising and head rotations showed significant differences across prosodic conditions, suggesting they are not merely byproducts of articulation but active visual cues.
  • Inter-Speaker Variation Exists: While general trends were observed, the study highlights that not all speakers utilize visual prosodic cues to the same degree or in the same way, indicating individual differences in visual communication strategies.

Key Results

  • Acoustic Correlates Confirmed: Auditory analysis confirmed that prosodic contrasts (duration, pitch, intensity) differed significantly across the three conditions, aligning with previous research.
  • Visual Movement Differences:
    • Articulatory: Narrow focus and echoic questions showed significantly greater jaw opening, lip opening, and jaw protrusion compared to broad focus statements.
  • Non-Articulatory: Brow raising and head rotations (pitch, roll) were significantly more pronounced for narrow focus and echoic questions relative to broad focus.
  • Variance Explained: The non-rigid PCs, guided by gPCA, accounted for over 91% of the facial movement variance for each speaker.
  • Statistical Significance: Many of the observed differences in visual parameters between prosodic conditions were statistically significant (p<.001).

Limitations and Future Work

  • Inter-Speaker Variation: The authors acknowledge that while general trends were found, individual speakers showed varying degrees of reliance on specific visual cues. Future work will involve recording more speakers to better understand these inter-speaker differences.
  • Temporal Relationship: The precise temporal relationship between auditory and visual prosodic cues is still under investigation. Future work aims to analyze this temporal synchronization to better understand the communicative function of visual prosody, potentially revealing how visual cues might act as pre-articulatory signals.
  • Focus on Critical Word: The analysis was primarily focused on the "critical word" for simplicity. Future work could explore how prosodic information is conveyed across entire utterances.
  • Generalization: While the study used a larger corpus and interactive task, further validation with even larger and more diverse datasets would strengthen generalization.

Summary updated

BibTeX
@inproceedings{cvejic2010prosody,
  author = {Erin Cvejic and Jeesun Kim and Chris Davis and Guillaume Gibert},
  title = {{Prosody for the eyes: quantifying visual prosody using guided principal component analysis}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure