Language specific effects of emotion on phoneme duration

Martijn Goudbeek, Mirjam BroersmaInterspeech 2010

TL;DR

This paper investigates how language and emotion interact to influence the duration of individual speech sounds. By analyzing a carefully constructed cross-linguistic corpus of emotional speech in Dutch and Korean, the researchers found that both language and emotion significantly alter phoneme durations, often in language-specific ways. This work highlights the complex interplay between universal emotional expression and language-specific phonetic realization.

What Problem Does This Paper Address?

While it's generally accepted that emotions can be recognized across cultures, the extent to which emotional expression is universal versus culturally or linguistically determined remains an active area of research. Previous studies have often used meaningless phrases or non-verbal vocalizations, or have not carefully controlled for linguistic compatibility in their stimuli. This has made it difficult to isolate the specific role of language in shaping how emotions are expressed phonetically. This paper tackles this by asking: do the specific sounds (phonemes) in our speech change duration differently when we express different emotions, and does this pattern vary between languages like Dutch and Korean?

Methods

The researchers developed a novel, cross-linguistically balanced corpus of emotional speech called the "Demo/Kemo corpus." This corpus features posed emotional expressions by both Dutch and Korean actors, using the same elicitation techniques and a carefully constructed carrier phrase, "[nuto hom sepikaŋ]," designed to be phonetically permissible and semantically neutral in both languages.

After recording the expressions, two judgment studies were conducted. First, native Dutch and Korean listeners rated the naturalness and identified the emotion of the recorded utterances. This allowed the researchers to select the most convincing portrayals for each actor-emotion pair.

Finally, the selected utterances were manually segmented at the phoneme level using the Praat speech analysis program. The duration of each individual phonetic segment within the carrier phrase was then extracted and analyzed. Statistical analyses, specifically mixed analyses of variance, were employed to determine the effects of language (Dutch vs. Korean) and emotion (Anger, Fear, Irritation, Sadness, Joy) on segment duration, as well as their interactions.

Datasets

The primary dataset introduced and utilized in this paper is the Demo/Kemo corpus. This corpus consists of:

  • Actors: 8 Dutch and 8 Korean professional actors.
  • Emotions: 8 emotions posed (though the study focuses on 5: Anger, Fear, Irritation, Sadness, and Joy).
  • Utterances: A fixed carrier phrase "[nuto hom sepikaŋ]" used for all expressions.
  • Listeners: Native Dutch and Korean listeners for judgment studies.

This corpus is notable for its balanced design, aiming to overcome limitations of previous cross-linguistic corpora by ensuring comparable elicitation, recording, and phonetic content across languages.

Key Insights and Ideas

  • Language Matters for Duration: The study strongly suggests that the duration of individual speech sounds is not solely dictated by emotion but is significantly influenced by the speaker's native language. Korean expressions were found to be shorter overall than Dutch ones.
  • Emotion's Subtle Hand: While there wasn't a universal main effect of emotion on segment duration across all sounds, the interaction between emotion and phonetic segment was significant. This means that how emotion affects duration is segment-specific.
  • Language-Emotion Interaction is Crucial: The most compelling finding is the significant three-way interaction between language, emotion, and phonetic segment. This indicates that the effect of emotion on phoneme duration is often different depending on the language.
  • Specific Segments are Affected: The influence of language and emotion on duration was not uniform across all sounds in the carrier phrase. Certain segments, particularly those in the middle and end of the utterance and sonorous ones, showed more pronounced effects.

Key Results

  • A main effect of language was found, with Korean expressions being shorter on average than Dutch expressions.
  • A main effect of phonetic segment was significant, indicating that segments naturally differ in duration.
  • A significant interaction between emotion and phonetic segment was observed, showing that emotions affect segment durations differently for different sounds.
  • A significant interaction between language and phonetic segment was also found, demonstrating language-specific patterns in segment durations.
  • The three-way interaction between language, emotion, and phonetic segment was highly significant (p < 0.001), underscoring the complex interplay.
  • For specific segments like /t/, /o/, /h/, and /ɔ/, emotion did not significantly affect duration. However, for segments like /n/, /p/, and /k/, emotion did affect duration, and this effect was often language-specific.

Limitations and Future Work

  • Limited Emotion Set: The study focused on five basic emotions. Future work could explore a wider range of emotions, including more nuanced or complex ones.
  • Limited Carrier Phrase: While carefully constructed, the analysis was limited to a single carrier phrase. Expanding to different phrases or naturalistic speech would provide broader insights.
  • Generalizability: The authors acknowledge that whether these specific effects generalize to other languages or other phonetic segments requires further investigation.
  • Cross-Cultural Decoding: While this study focused on expression, future research could investigate how these language-specific duration patterns affect the decoding of emotions by listeners from different linguistic backgrounds.
  • Other Phonetic Features: This study focused on duration. Future work could explore how language and emotion interact to influence other phonetic features like pitch or intensity.

Summary updated

BibTeX
@inproceedings{goudbeek2010language,
  author = {Martijn Goudbeek and Mirjam Broersma},
  title = {{Language specific effects of emotion on phoneme duration}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure