Towards an Annotation Scheme for Complex Laughter in Speech Corpora

Khiet P. Truong, Jürgen Trouvain, Michel-Pierre JansenInterspeech 2019

TL;DR

This paper tackles the challenge of consistently annotating complex human laughter in speech, which is crucial for developing more sophisticated speech recognition and human-computer interaction systems. They propose a multi-level annotation scheme and evaluate its effectiveness across various corpora, highlighting significant differences in annotation agreement based on whether laughter is posed or spontaneous.

What Problem Does This Paper Address?

Laughter is a fundamental aspect of human communication, yet its annotation in speech corpora is surprisingly inconsistent. Existing research lacks a standardized approach, leading to difficulties in comparing studies and building reliable automatic laughter detection systems. This inconsistency is particularly pronounced when dealing with "complex laughter," which goes beyond simple, isolated vocalizations and includes elements like speech-laughs (laughter intertwined with speech), smiled speech, and intricate respiratory patterns. The authors aim to bridge this gap by developing and testing a detailed annotation scheme that accounts for these complexities, ultimately seeking to improve the homogeneity and transparency of laughter annotation.

Methods

The core of this paper is the proposed six-level annotation scheme for laughter. This scheme breaks down laughter into increasingly granular components:

  • Level 1 (SPEECH-RESP): Differentiates between speech (SP) and respiratory noises (inbreath/outbreath - INBR/OUTBR), which can serve as onsets or offsets for laughter.
  • Level 2 (EPISODE): Defines a complete laugh event, encompassing one or more "bouts."
  • Level 3 (BOUT): Identifies a sequence of laugh-like syllables within a single exhalation phase. This level also includes "onset" and "offset" markers, as well as "speech-laughs" (SPL) and "smiled speech" (SMSP).
  • Level 4 (SUB-BOUT): Further categorizes bouts into voiced (VCD), unvoiced (UVD), or silent (SIL) segments.
  • Level 5 (SEGMENTAL): Distinguishes between voiced pulses (VP) and unvoiced pulses (UP) within sub-bouts.
  • Level 6 (Optional - AUDIBLE SMILING): Specifically annotates audible smiling, distinct from speech-laughs.

The authors applied this scheme to several corpora, with two experienced annotators working independently. They then analyzed inter-annotator agreement using metrics like the percentage of matched units and the average misaligned time for matched units.

Datasets

The study utilized six different corpora to ensure a diverse range of laughter types and communicative situations:

  • Montreal Affective Voices [28]: Posed laughter from actors.
  • AVLaughterCycle [29]: Laughter evoked by watching video clips.
  • GECO [30]: German free conversation.
  • Lindenstrasse [31]: German task-based dialogues related to a TV soap opera.
  • ICSI [32]: English multi-party meetings.
  • Mulai [11]: English task-based dialogues where participants try to make each other laugh.

For each corpus, 10 audio samples were randomly selected, resulting in a total of 120 annotation files for analysis.

Key Insights and Ideas

  • Posed vs. Spontaneous Laughter: A major finding is the significant difference in annotation agreement between posed laughter (like that in the Montreal corpus) and spontaneous, social laughter. Posed laughter is more homogeneous and easier to annotate consistently.
  • Complexity of Spontaneous Laughter: Spontaneous laughter, especially in conversational settings, is far more complex, featuring more onsets and offsets, and a greater prevalence of unvoiced segments and speech-laughs.
  • Importance of Respiratory Noises: Respiratory noises (inbreaths and outbreaths) play a more critical role in spontaneous laughter than previously acknowledged, often marking episode onsets or offsets.
  • Challenges with Speech-Laughs and Smiled Speech: Distinguishing between speech-laughs and smiled speech, and determining their role as linking elements within an episode, proved to be a significant source of disagreement among annotators.
  • Episode Boundary Ambiguity: Even with high temporal overlap, annotators often disagreed on the precise start and end boundaries of laughter episodes, particularly when a single event could be interpreted as one or multiple distinct episodes.

Key Results

  • High Agreement on Episodes: Annotators showed good agreement on identifying laughter episodes across most corpora.
  • Lower Agreement on Lower Levels: Agreement dropped significantly for lower-level units like bouts, sub-bouts, and especially for specific categories like speech-laughs (SPL) and smiled speech (SMSP).
  • Montreal Corpus Outlier: The Montreal corpus (posed laughter) consistently showed higher inter-annotator agreement and lower misaligned times compared to other corpora.
  • Misalignment Varies by Level: Misalignment was lowest for episodes and segmental levels, and highest for bout and sub-bout levels. Unvoiced pulses also showed greater temporal misalignment than voiced pulses.
  • Specific Disagreements: ONOFFSET (in/outbreath) and the distinction between SPL and SMSP were identified as key areas of annotator confusion.

Limitations and Future Work

  • Annotation Scheme is a Draft: The authors acknowledge that their proposed scheme is a first attempt and should be viewed as a draft, with the hope of it evolving into a standard.
  • Audio-Only Focus: The current scheme is primarily designed for audio data, deliberately excluding visual information. Future work could explore how integrating visual cues might improve annotation, especially for complex laughs.
  • Further Investigation Needed: More research is needed to better understand and annotate complex laughter phenomena like short silent intervals, speech-laughs, and smiled speech, particularly their role in linking laughter elements.
  • Corpus-Specific Differences: The significant differences observed between corpora highlight the need for annotation schemes that can adapt to or account for variations in laughter production and context.

Summary updated

BibTeX
@inproceedings{truong2019towards,
  author = {Khiet P. Truong and Jürgen Trouvain and Michel-Pierre Jansen},
  title = {{Towards an Annotation Scheme for Complex Laughter in Speech Corpora}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure