Exploratory Study of Some Acoustic and Articulatory Characteristics of Sad Speech

Donna Erickson, Kenji Yoshida, Caroline Menezes, Akinori Fujino, Takemi Mochida, Yoshiho ShibuyaPhonetica 2006 · DOI

TL;DR

This study investigates the acoustic and articulatory features of spontaneous sad speech in American English and Japanese. It reveals that while acoustic cues like pitch and voice quality are similar between spontaneous and imitated sadness, the articulatory movements of the mouth and tongue differ significantly, suggesting that "conveying" sadness through speech involves different mechanisms than experiencing it.

What Problem Does This Paper Address?

Communicating emotion through speech is a complex interplay of linguistic content and paralinguistic cues. While much research has focused on identifying acoustic correlates of emotions, there's a significant gap in understanding how spontaneous emotional speech, as opposed to acted or imitated speech, is produced and perceived. Specifically, this paper aims to differentiate the acoustic and articulatory characteristics of genuine sadness from imitated sadness and read speech, exploring whether these differences are language-dependent and how they relate to listener perception. The authors highlight the importance of distinguishing between "acted" emotions (controlled by the speaker for effect) and "spontaneous" emotions (arising from internal states), which may involve involuntary physiological changes.

Methods

The study employed a multi-modal approach, combining acoustic and articulatory measurements.

  • Articulatory Data: 2D Electromagnetic Articulography (EMA) was used to track the precise movements of the lips, jaw, and tongue. This technique involves attaching small sensors to the articulators and recording their positions in 3D space.
  • Acoustic Data: Standard acoustic analysis techniques were used to measure fundamental frequency (F₀), formant frequencies (F₁, F₂), duration, and voice quality (e.g., glottal opening quotient (AQ) for American English, spectral tilt for Japanese).
  • Experimental Design: Two female speakers (one American English, one Japanese) were recorded under several conditions:
  • Spontaneous Sad Speech (E): Recorded during informal conversations where speakers were experiencing genuine sadness (due to personal loss).
  • Imitated Sad Speech (IE): Speakers imitated their own spontaneous sad utterances, including phrasing, intonation, and emotion.
  • Imitated Intonation (II) (American English only): Speakers imitated only the phrasing and intonation patterns of their spontaneous sad utterances, without the emotion.
  • Read Speech (R): Speakers read the original utterances.
  • Perceptual Analysis: Listeners rated the perceived degree of sadness for each utterance on a 5-point scale.

Datasets

The study utilized two primary datasets, each collected from a single female speaker:

  • American English:
    • Spontaneous Sad Speech: Recorded during informal conversations related to a mother's terminal illness and death.
  • Control Data: Imitated sad speech, imitated intonation speech, and read speech of the same utterances.
  • Japanese:
    • Spontaneous Sad Speech: Recorded during informal conversations following the loss of a mother due to a brain aneurysm.
  • Control Data: Imitated sad speech and read speech of the same utterances.

The specific utterances analyzed were the word "leave" for the American English speaker and "kara" for the Japanese speaker, chosen because they occurred during moments of strong sad emotion.

Key Insights and Ideas

  • Spontaneous vs. Imitated Sadness: While acoustic cues like high F₀ and altered voice quality were similar between spontaneous and imitated sad speech, the articulatory movements (lip, jaw, and tongue positions) differed significantly. This suggests that the physical act of experiencing sadness leads to different articulatory configurations than mimicking sadness.
  • "Conveying" Sadness: Well-perceived sad speech, as rated by listeners, often involved lip-pouting and a raised/protruded jaw/lower lip. This articulatory pattern was more similar to imitated sad speech than to spontaneous sad speech, implying that the articulatory features that "convey" sadness to a listener might be different from those produced during genuine emotional distress.
  • Role of Intonation: Imitated intonation alone was not sufficient to convey sadness, highlighting the importance of other acoustic cues like F₀ and voice quality.
  • Language Similarities: Despite linguistic differences, both American English and Japanese speakers showed common acoustic tendencies (raised F₀, changed F₁, voice quality changes) and articulatory tendencies (retracted lips, lowered jaw) in spontaneous sad speech.
  • Physiological vs. Stereotypical Production: The study speculates that spontaneous sadness, especially when accompanied by crying, might involve involuntary physiological changes affecting articulation. In contrast, imitated sadness might rely more on learned stereotypes of what sad speech sounds like.

Key Results

  • Perception: Listeners ranked both spontaneous and imitated emotional speech as sadder than read speech. For American English, imitated intonation speech was rated as less sad than spontaneous or imitated sad speech.
  • Acoustics:
    • F₀: Both speakers showed significantly higher F₀ in sad and imitated sad speech compared to read speech.
  • Voice Quality: American English speakers showed lower AQ (indicating less breathiness) in sad and imitated sad speech. Japanese speakers exhibited a steep spectral slope in imitated sad speech.
  • Duration: Averaged durations of imitated sad speech were generally longer than spontaneous sad speech for the American English speaker.
  • Articulation:
    • American English: Spontaneous sad speech involved retracted and raised lips, a lowered jaw, and fronted/raised tongue parts. Imitated sad speech showed protruded lips and a protruded/raised jaw.
  • Japanese: Spontaneous sad speech involved retracted and lowered lips, a protruded and lowered jaw, and a fronted tongue tip. Imitated sad speech showed protruded and raised lips, and a retracted and raised jaw.
  • Correlation: For well-perceived sad speech, listener ratings correlated significantly with acoustic features like high F₀ and changed voice quality. Articulatory features of imitated sad speech showed a better correlation with listener ratings than spontaneous sad speech.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Sample Size: The study is based on only two speakers (one for each language), limiting the generalizability of the findings.
  • Non-Laboratory Design: The linguistic content and timing of emotional expression were not strictly controlled, making it difficult to isolate specific variables.
  • Bimodal Distribution: The American English speaker's data showed a bimodal distribution in some analyses, which might be an artifact of the small sample size or variations in F₀ production.

Future work suggested includes:

  • Larger Speaker Samples: Expanding the study to include more speakers from diverse linguistic backgrounds.
  • Parametric Factorization: Investigating underlying dimensions of genuine and imitated emotion through more advanced analysis techniques.
  • Articulatory-Acoustic Links: Further exploring the articulatory underpinnings of voice quality changes associated with emotion.
  • Cross-Linguistic Comparisons: Deeper investigation into commonalities and differences in emotional speech production and perception across languages.

Summary updated

BibTeX
@article{erickson2006exploratory,
  author = {Donna Erickson and Kenji Yoshida and Caroline Menezes and Akinori Fujino and Takemi Mochida and Yoshiho Shibuya},
  title = {{Exploratory Study of Some Acoustic and Articulatory Characteristics of Sad Speech}},
  journal = {Phonetica},
  year = {2006},
  doi = {10.1159/000091404},
}
Figure