Speaker Movement Correlates with Prosodic Indicators of Engagement

Rob Voigt, Robert J. Podesva, Dan Jurafsky2014 · DOI

TL;DR

This paper introduces a novel, automatic method for quantifying speaker body movement from video, demonstrating that increased movement amplitude is strongly correlated with higher and more variable pitch and intensity in speech. This work bridges the gap between acoustic and visual prosody, offering a scalable way to study embodied engagement in naturalistic speech.

What Problem Does This Paper Address?

Understanding how our bodies contribute to communication, particularly how movement and gesture interact with speech prosody (the rhythm, stress, and intonation of speech), is a complex challenge. While previous research has shown correlations between discrete gestures and specific prosodic events like pitch accents, there's a significant gap in understanding how continuous body movements relate to broader indicators of speaker engagement. Existing methods for analyzing movement are often manual, time-consuming, and require specialized equipment, limiting the scale of research. This paper aims to address this by developing an automatic, replicable method to quantify continuous body movement and investigate its relationship with prosodic features indicative of engagement.

Methods

The authors introduce a new metric called Movement Amplitude (MA). This metric quantifies continuous body movement from video by measuring frame-to-frame visual changes. Specifically, it calculates the average difference in RGB values between corresponding pixels in successive frames, sums these differences across the entire image, and then takes the natural logarithm. This approach is fully automatic and requires only standard video input.

For acoustic analysis, they extract fundamental frequency (pitch) and intensity features for each automatically identified Pause-Bounded Unit (PBU). PBUs are segments of speech separated by pauses, identified using a heuristic algorithm in the Praat software. For each PBU, they calculate the maximum, mean, minimum, and standard deviation for both pitch and intensity.

To handle the high collinearity of movement amplitude measures, they employ Principal Component Analysis (PCA) to reduce dimensionality, identifying two main components: "overall movement" and "movement variance." Statistical analysis is performed using linear mixed-effects models, treating speakers as random effects to account for individual differences.

Datasets

The study uses a newly collected dataset of 14 YouTube "First Day of School" vlogs, totaling 95 minutes of footage. These videos feature female, high-school-aged students from the United States speaking into their cameras against static backgrounds. The authors chose this genre and demographic to control for potential sociolinguistic differences and focus on the core relationship between movement and prosody.

Key Insights and Ideas

  • Movement as a Continuous Indicator of Engagement: The paper moves beyond discrete gestures to show that the amount of continuous body movement is a significant correlate of speaker engagement, as reflected in prosodic features.
  • Bridging the Visual-Prosody Gap: MA provides a quantitative link between visual "prosody" (body movement) and acoustic prosody, supporting the idea that these are integrated aspects of communication.
  • Scalability and Replicability: The automatic nature of MA is a major contribution, enabling large-scale analysis of "in-the-wild" video data, which was previously infeasible.
  • Movement Variance vs. Overall Movement: Interestingly, while overall movement amplitude was strongly predictive of prosodic features, movement variance (fluctuations in movement within a phrase) was not. This suggests that the degree of movement, rather than its variability, is more directly tied to engagement indicators in speech.

Key Results

  • High overall movement amplitude (Factor 1 from PCA) was statistically significantly associated with increased pitch maximum, mean, and standard deviation, as well as intensity maximum, mean, and standard deviation.
  • For example, a one standard deviation increase in overall movement predicted an increase in pitch mean by 2.797 Hz and intensity mean by 0.152 dB.
  • Movement variance (Factor 2 from PCA) did not show a significant predictive relationship with any of the measured acoustic features.
  • The study found that phrases with higher movement amplitude also tended to have higher values for acoustic prosody measures, visualized effectively in Figure 3.

Limitations and Future Work

  • Coarseness of MA: The current MA measure captures all movement (eyes, mouth, head, body) and compresses 3D movement into a 2D representation. It is also limited by the need for static backgrounds and visually separable speakers.
  • Future Work on MA Granularity: The authors suggest that future work could refine MA by using computer vision techniques (like face detection) to isolate specific body parts (e.g., facial movement vs. body movement) or to separate semantic gestures from general movement.
  • Exploring Movement Variance: The lack of correlation for movement variance suggests that temporal and local synchrony between gesture apices and pitch accents might be more important than the variability of movement itself. This warrants further investigation.
  • Broader Applications: The authors plan to expand this research to include genre effects, social meaning, contextual factors (like performativity), conversational speech, and sociolinguistic variables (age, gender, dialect).

Summary updated

BibTeX
@misc{voigt2014speaker,
  author = {Rob Voigt and Robert J. Podesva and Dan Jurafsky},
  title = {{Speaker Movement Correlates with Prosodic Indicators of Engagement}},
  year = {2014},
  doi = {10.21437/speechprosody.2014-2},
}
Figure