The Role of Creaky Voice in Turn Taking and the Perception of Speaker Stance: Experiments Using Controllable TTS

Harm Lameris, Eva Szekely, Joakim GustafsonCOLING 2024

TL;DR

This paper explores how synthesized creaky voice, a non-standard vocal quality, affects listener perceptions of speaker certainty, valence, and turn-taking cues. By developing a controllable Text-to-Speech (TTS) system, they demonstrate that different types of creaky voice can subtly alter how listeners interpret a speaker's stance and their intention to yield the conversational floor.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems are increasingly capable of generating natural-sounding speech, but they often lack the nuanced vocal qualities that humans use to convey complex pragmatic and paralinguistic information. Creaky voice, characterized by irregular glottal vibration, is one such quality that humans use to signal things like uncertainty, detachment, or the end of a conversational turn. However, understanding its precise perceptual impact, especially when synthesized, and how to control it effectively in TTS remains an open challenge. This paper aims to bridge this gap by investigating how listeners perceive synthesized creaky voice in terms of speaker stance and turn-finality.

Methods

The researchers employed a two-pronged approach: objective analysis and subjective listening experiments.

  • Creaky Voice Synthesis: They modified a Tacotron2-based neural TTS system. This involved:
  • Annotating a spontaneous speech corpus for creaky voice using two automatic detection tools (DeepFry and CreaPy).
  • Developing a "creak embedding" that encodes the per-word creak percentage. This embedding is appended to the model's input, allowing for controllable synthesis of creaky voice.
  • Training the model on both read and spontaneous speech data, with a specific focus on incorporating the creak embedding during fine-tuning.
  • Objective Evaluation: They used the CreaPy tool to quantify the actual creak percentage in the synthesized stimuli across different phonation types (modal, non-positional creak, positional creak).
  • Subjective Listening Experiments: Two experiments were conducted with non-expert listeners:
  • Non-Positional Creak Experiment: Participants rated synthesized phrases where non-positional creak was introduced to simulate parenthetical comments or ambiguity.
  • Positional Creak Experiment: Participants rated synthesized phrases where positional creak was used, typically at the end of an utterance, to signal turn-finality.
  • Listeners rated stimuli on a 7-point scale for certainty, valence, sarcasm, and turn finality.

Datasets

  • Training Data:
    • A spontaneous speech corpus from the publicly available multimodal multi-party dataset "AptSpeech" (Kontogiorgos et al., 2018). This corpus consists of interactions between a mediator and participants designing an apartment.
  • 1129 CMU Arctic sentences and 1132 sentences from online newspaper texts, read aloud.
  • Evaluation Data: The synthesized stimuli generated by their modified TTS system.

Key Insights and Ideas

  • Synthesized Creak is Perceptible: Listeners without formal phonetic training can reliably perceive synthesized creaky voice, suggesting the TTS system effectively captures this vocal quality.
  • Creaky Voice Modulates Perceptions: Different types of creaky voice have distinct perceptual effects, influencing how listeners interpret speaker certainty, valence, and turn-finality.
  • Non-Positional Creak Signals Uncertainty and Detachment: Non-positional creak was perceived as less certain and less positive than modal voice, aligning with its use as a hedging or distancing device.
  • Positional Creak Signals Turn Finality: Positional creak was strongly perceived as indicating the end of a turn, reinforcing its role as a prosodic cue for conversational flow.
  • Sarcasm is Not Solely Cued by Creak: While creaky voice might accompany sarcasm, the study found no significant difference in sarcasm ratings between modal and creaky voices, suggesting other cues are more dominant.

Key Results

  • Objective Creak Measurement: The objective evaluation confirmed significant differences in creak percentage between modal, non-positional creak, and positional creak stimuli. Positional creak had the highest percentage, followed by non-positional creak, and then modal voice.
  • Non-Positional Creak Perception:
    • Certainty: Creaky stimuli were rated as less certain (median 5 vs. 5 for no creak, but with a significant difference, p=.04).
  • Valence: Creaky stimuli were rated as more negative (median 4 vs. 4 for no creak, but with a significant difference, p=.0002).
  • Turn Finality: Creaky stimuli were rated as more turn final (median 6 vs. 6 for no creak, with a significant difference, p=.008).
  • Positional Creak Perception:
    • Turn Finality: Positional creak was rated significantly more turn final than modal phonation (median 6 vs. 6, p<.0001).
  • No significant differences were found for certainty, valence, or sarcasm in the positional creak experiment.

Limitations and Future Work

  • Limitations:
    • The study focused on a single speaker's voice (the mediator) and a specific context (conversational task).
  • While creaky voice decreased certainty and valence, the median ratings for these attributes remained positive, indicating that creaky voice doesn't automatically imply extreme uncertainty or negativity.
  • The findings on sarcasm suggest creak is not the primary cue, but further investigation is needed.
  • Future Work:
    • Investigate the link between sarcasm, irony, and creak, and how it interacts with other creaky voice indexations.
  • Explore the sociolinguistic aspects of creaky voice perception, particularly concerning gender.
  • Further research into the precise function of creak in signaling turn yields, especially in cross-linguistic contexts.

Summary updated

BibTeX
@inproceedings{lameris2024role,
  author = {Harm Lameris and Eva Szekely and Joakim Gustafson},
  title = {{The Role of Creaky Voice in Turn Taking and the Perception of Speaker Stance: Experiments Using Controllable TTS}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure