Vocal tract contour analysis of emotional speech by the functional data curve representation

Sungbok Lee, Shrikanth S. NarayananInterspeech 2010 · DOI

Introduction

While the emotional content of speech is often studied through acoustic features like pitch and energy, these sounds are the result of physical changes in the vocal tract (VT). Understanding the relationship between emotional state and the physical geometry of speech organs—such as the tongue and lips—remains a significant challenge due to anatomical variability and the difficulty of real-time imaging. Researchers at the University of Southern California (USC) have addressed this by combining real-time MRI (rtMRI) with Functional Data Analysis (FDA) to create a robust mathematical framework for analyzing emotional articulation.

Methodology: From MRI to Mathematical Curves

The study utilized a custom rtMRI setup to capture the midsagittal vocal tract at 21 frames per second. Two native English speakers (one male, one female) produced sentences across four emotional states: Neutral, Angry, Sad, and Happy.

To process this data, the researchers employed a region-based automatic segmentation algorithm to trace the contours of the lower vocal tract, focusing on the tongue and lips.

Vocal tract contours showing variations in tongue and lip positions across different emotional states.

The core innovation lies in the use of Functional Data Analysis (FDA). Rather than treating the vocal tract contour as a set of discrete points, FDA represents the entire curve as a continuous function using 16 B-spline basis functions. This approach allows for anatomical normalization; by using a fixed number of coefficients, researchers can compare articulatory "gestures" across different speakers regardless of their specific vocal tract length or shape.

Key Insights: Articulatory Signatures of Emotion

The analysis revealed that emotions significantly alter the "baseline" posture of the vocal tract. For instance, "sad" speech was often characterized by a lower tongue configuration compared to neutral speech. Interestingly, the physical expression of emotion appeared to be speaker-dependent. While one subject showed distinct variations in the pharyngeal (back) region during anger, the other exhibited more variation in the front (lips and tongue blade).

By converting these physical shapes into a compact set of 16 FDA coefficients, the researchers created a feature vector that serves as a unique "articulatory signature" for each emotion. This mathematical representation is robust against anatomical differences, making it a promising tool for general-purpose speech production research.

Results and Classification

Using Fisher Linear Discriminant Analysis (LDA), the system achieved a 65.0% cross-validation accuracy in discriminating between the four emotions. This performance is notable as it rivals results typically obtained from much larger sets of acoustic features, suggesting that articulatory data is a highly efficient indicator of emotional state.

FDA coefficient profiles for two subjects across four emotions, highlighting distinct articulatory signatures and speaker-dependent variations.

The data showed significant variation in "articulatory expressiveness" between subjects. As seen in the FDA coefficient profiles, Subject AB’s patterns were more distinct (achieving 75% accuracy for x-coordinates) compared to Subject EA (60% accuracy). This suggests that some individuals may rely more heavily on physical articulatory changes to convey emotion, while others might prioritize acoustic cues.

Conclusion and Future Directions

This study provides a proof-of-concept for using FDA to normalize and analyze complex vocal tract movements. While limited by a small sample size of two speakers, the results demonstrate that the physical geometry of speech carries substantial emotional weight. Future work aims to expand the cohort and integrate these articulatory features with traditional prosodic data to develop a more holistic model of emotional speech production.

Summary updated

BibTeX
@inproceedings{lee2010vocal,
  author = {Sungbok Lee and Shrikanth S. Narayanan},
  title = {{Vocal tract contour analysis of emotional speech by the functional data curve representation}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-464},
}
Figure