A detailed study of word-position effects on emotion expression in speech

Jangwon Kim, Sungbok Lee, Shrikanth S. NarayananInterspeech 2009 · DOI

Abstract

This research explores the relationship between word position within an utterance and the intensity of emotional expression. Using the Electromagnetic Articulography (EMA) database, the study analyzes both acoustic and articulatory features across different sentence positions. Findings demonstrate that "sentence-medial" words carry significantly stronger emotional signatures than initial or final words. Articulatory movement data proved to be a highly potent carrier of emotion, with recognition rates reaching 100% for medial words, suggesting that word-position sensitivity can significantly enhance automatic emotion recognition (AER) systems.

Introduction

Most automatic emotion recognition (AER) systems treat an entire utterance as a single emotional unit, ignoring the inherent heterogeneity of human speech. While prosody and rhythm vary across a sentence, the impact of word location on "emotional loading" has remained under-explored. This paper investigates whether emotion is expressed more vividly in specific sentence segments, offering a pathway to more robust AER systems. The authors hypothesize that emotional expression is not uniform and that certain positions within an utterance may serve as "sweet spots" for emotional information.

Methods

The researchers employed a dual-track approach, analyzing both acoustic signals and physical articulatory movements. The study utilized the EMA (Electromagnetic Articulography) database, consisting of 680 utterances from three native speakers of American English across four emotions: Neutral, Angry, Sad, and Happy.

Articulatory Analysis

Seven specific kinematic points, including closures and releases of the tongue tip and lower lip, were manually labeled for target words in various sentence positions to track physical movement range and velocity.

Articulatory trajectories (tongue tip and lower lip) synchronized with the acoustic waveform.

Modeling and Classification

Gaussian Mixture Models (GMMs) were trained to classify speech as either "Neutral" or "Emotional." Acoustic features included 12th-order Mel-Frequency Cepstral Coefficients (MFCCs) and Mel Filter Banks (MFBs). Articulatory features consisted of position and velocity data for the tongue tip, lower lip, and jaw, reduced via Principal Component Analysis (PCA).

Key Results and Insights

The Medial Position Advantage

The study found that emotional effects are most pronounced in the middle of a sentence. For instance, the word "FOAM" saw its emotion recognition rate jump from 64% in the initial position to 89% in the medial position when using MFBs. Statistical analysis via ANOVA confirmed that kinematic features are significantly more sensitive to emotional stress in medial positions.

ANOVA results comparing the statistical significance of kinematic features in initial vs. medial positions.

As shown in the table above, almost all duration-related features of tongue movement reached statistical significance ($p < 0.05$) in medial positions, whereas initial positions showed significance only for a limited subset of features like velocity and total duration.

Articulatory vs. Acoustic Performance

Models trained on physical movement data achieved near-perfect recognition rates (up to 100% for some medial words), significantly outperforming acoustic-only models. This suggests that the physical "body language" of the vocal tract is an incredibly potent carrier of emotion. Furthermore, Mel Filter Banks (MFBs) generally outperformed MFCCs in word-level recognition, likely because they preserve more raw spectral information altered by emotion.

Discussion and Limitations

While emotional states lead to faster and more extreme articulatory movements, the timing between different articulators (articulatory phase) remains relatively stable. This indicates that speakers maintain fundamental linguistic coordination even under emotional stress. The "position effect" was particularly evident in single-syllable words; multi-syllable words like "TANTAMOUNT" provided more "surface area" for emotional expression, maintaining high performance across positions.

The study's primary limitations include a small sample size of three speakers and the reliance on manual articulatory labeling. Future work should focus on developing automatic segmentation techniques to scale this research to larger, more diverse datasets.

Conclusion

Word position is a critical factor in the expression of emotion in speech. By accounting for these position-dependent "emotional loadings," future AER systems can be optimized to focus on high-information segments, leading to improved accuracy and robustness in real-world applications.

Summary updated

BibTeX
@inproceedings{kim2009detailed,
  author = {Jangwon Kim and Sungbok Lee and Shrikanth S. Narayanan},
  title = {{A detailed study of word-position effects on emotion expression in speech}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-481},
}
Figure