Effects of emotion on the lower lip movements at phrase boundaries

Sungbok Lee, Shrikanth Narayanan2012 · DOI

Abstract

In this preliminary report, we investigate the effects of emotion on lower lip movements during the production of a monosyllabic word in two different prosodic contexts: phrase-initial and phrase-final. The purpose is to examine the rigidity of phrase boundary effects under emotional perturbation of neutral speech production. It is found that the effect of emotion is ubiquitous, affecting all kinematic and dynamic parameters of lower lip movements. However, these effects are not arbitrary. Speakers maintain phrase boundary effects on duration and movement amplitude under emotional variations. Furthermore, a linear relationship between movement amplitudes and maximum lip opening velocities suggests that the stiffness of lower lip opening gestures is almost invariant across emotions, implying regulation by a simple harmonic oscillator-like system. These findings support the hypothesis that speech production constraints imposed by phrase boundaries are maintained against emotional perturbations.

Introduction

Speech is a complex, multi-layered signal where structural linguistic rules (prosody) and affective states (emotion) coexist. Prosody provides the structural rhythm of language, such as the natural lengthening of words at the end of a sentence (phrase-final lengthening). Emotion, conversely, colors articulation with varying degrees of arousal and valence. A central question in speech science is how these systems interact: does intense emotion override the mechanical rules of phrase boundaries, or does the physical "skeleton" of speech remain rigid? This study investigates the kinematics and dynamics of the lower lip to determine which layer takes precedence during emotional speech.

Methodology

The study employed Electromagnetic Articulography (EMA) to track speech movements with high precision. Three professional actors (one male, two female) participated, with six sensors attached to the tongue, lips, and jaw. The task involved speaking seven neutral sentences, each repeated five times across five emotional states: neutral, hot anger, cold anger, sadness, and happiness.

The analysis focused on the monosyllabic word "FIVE," positioned in two contexts: phrase-initial and phrase-final. Using the "mview" MATLAB tool, researchers measured duration, movement amplitude, and maximum velocity. "Stiffness" was calculated as the ratio of maximum velocity to movement amplitude. To ensure emotional authenticity, recordings were validated by five native English listeners; only tokens with high identification rates were analyzed (e.g., 34 tokens for subject JN and 38 for subject JR).

Results

The data revealed that while emotion significantly impacts the duration and speed of lip movements, it does not erase the underlying prosodic structure.

Persistence of Phrase Boundary Effects

A key finding was the consistent maintenance of phrase-final lengthening across all emotional states. As shown in Figure 5, phrase-final gestures remained distinct from phrase-initial gestures regardless of whether the speaker was expressing anger, happiness, or sadness.

Duration of lower lip gestures across emotions and phrase positions for subjects JN and JR.

Statistical analysis confirmed that phrase-final words were significantly longer than phrase-initial words ($p < .001$ for both subjects). While emotion influenced the absolute duration, the relative difference between phrase positions remained stable.

The Harmonic Oscillator Model

The study suggests that the lower lip gesture is regulated by a system resembling a simple harmonic oscillator. For each speaker, an approximately linear relationship was observed between movement amplitude and maximum velocity across different emotions. This implies that the "stiffness" of the lip—analogous to the tension in a mechanical spring—is nearly invariant under emotional perturbation.

Vertical movement trajectories of the lower lip, illustrating the consistency of gesture patterns under emotional perturbation.

Figure 6 illustrates the vertical movement trajectories, showing how the basic shape and timing of the gesture are preserved. For subject JN, the effect of the phrase boundary on stiffness was notably stronger than the effect of emotion ($F=45.04$ vs $F=6.64$), further highlighting the rigidity of the linguistic structure.

Discussion and Conclusion

The findings indicate that the physical changes in lip movement are primarily driven by the arousal dimension of emotion. High-arousal states like "hot anger" produced the fastest and most expansive movements, while low-arousal states like "sadness" resulted in the weakest. Despite these variations, the underlying motor control system preserves linguistic constraints.

Limitations

As a preliminary report, the study is limited by a small sample size (three actors) and a restricted vocabulary (the word "FIVE"). Future research is required to determine if these harmonic oscillator rules apply to other articulators, such as the tongue, and across a wider variety of phonemes.

In conclusion, the speech motor control system appears to have a "base setting" for linguistic structure that remains remarkably resilient to emotional interference, ensuring that the structural integrity of language is maintained even in highly affective communication.

Summary updated

Abstract

In this preliminary report we investigate the effects of emotion on the lower lip movements during the production of a monosyllabic word in two different prosodic contexts: phrase initial and phrase final. The purpose is to examine the rigidity of the phrase boundary effects under emotional perturbation of the normal, or neutral, speech production. It is found that the effect of emotion is ubiquitous in that it affects all kinematic and dynamic parameters of the lower lip movements considered in this study. The effects of emotion are not arbitrary, however, and some common characteristics of emotional speech articulation can be identified. Firstly, speakers maintain the phrase boundary effects on the duration and movement amplitude of the lower lip gesture under emotional variations. Secondly, when considering all emotion types together in a given speaker, an approximately linear relationship holds between movement amplitudes and maximum lip opening velocities when emotion varies. Therefore, the stiffness of the lower lip opening gestures seems almost invariant under emotional variations, implying that the lower lip gesture is regulated by a simple harmonicoscillator like system with a constant stiffness. These and other findings support the hypothesis that speech production constraints imposed by the phrase boundary condition are maintained against emotional perturbations of speech articulations. In addition, the findings in the study may point to a general emotional speech production mechanism that the effects of emotion on the kinematics and dynamics of the oral speech articulators are manifested mainly in the arousal dimension of emotion.
BibTeX
@misc{lee2012effects,
  author = {Sungbok Lee and Shrikanth Narayanan},
  title = {{Effects of emotion on the lower lip movements at phrase boundaries}},
  year = {2012},
  doi = {10.21437/speechprosody.2012-115},
}
Figure