Sincerity in Acted Speech: Presenting the Sincere Apology Corpus and Results

Alice Baird, Eduardo Coutinho, Julia Hirschberg, Björn W. SchullerInterspeech 2019

TL;DR

This paper introduces the Sincere Apology Corpus (SINA-C), a new dataset of acted apologies, and presents baseline results for automatically detecting sincerity in speech. They found that deep learning representations, particularly spectrogram-based features, outperform traditional acoustic features for this task, highlighting the potential for AI to understand nuanced human emotions in speech.

What Problem Does This Paper Address?

Discerning sincerity in speech is crucial for human interaction, influencing trust and communication. As interactions increasingly move online, the ability to automatically detect sincerity from speech becomes vital for applications ranging from customer service to online banking fraud detection. While deception detection in speech has been a focus, sincerity detection, especially in the context of apologies, is less explored. This paper addresses this gap by creating a dedicated corpus of apologies and developing computational models to predict sincerity levels.

Methods

The researchers employed a two-pronged approach to sincerity detection: classification (sincere vs. not sincere) and regression (degree of sincerity). They experimented with three main types of acoustic features:

  • Hand-crafted features:
    • COMPARE: A large set of 6329 acoustic features.
  • EGEMAPS: A more compact set of 88 acoustic features.
  • Deep learning features:
    • DEEP SPECTRUM toolkit: This approach uses pre-trained Convolutional Neural Networks (CNNs) to extract visual representations from audio signals (spectrograms and mel-spectrograms). They extracted features from the activations of the network's fully connected layers (fc6 and fc7).

For modeling, they used Support Vector Machines (SVM) for classification and linear Support Vector Regression (SVR) with linear kernels. They explored two training/validation schemas: Speaker Independent Folds (C-SIF) and a 3-fold Speaker Independent Nested Cross Validation (SICV).

Datasets

The primary dataset introduced and used in this paper is the Sincere Apology Corpus (SINA-C).

  • SINA-C: This is an English acted-speech corpus consisting of 912 audio instances from 32 speakers (15 male, 17 female, aged 20-60). The speakers delivered apologies in various prosodic styles (monotonic, stress, fast rate, slow rate) and six different apology formulations. The corpus was annotated by 22 individuals for perceived sincerity on a 5-point Likert scale, which was then standardized and converted into binary (Sincere/Not Sincere) and continuous (degree of sincerity) labels. SINA-C is available as an open-access dataset.

Key Insights and Ideas

  • Deep representations are key: The study strongly suggests that deep learning-based features (from DEEP SPECTRUM) are superior to traditional hand-crafted acoustic features for sincerity detection. This indicates that deep models can capture more subtle, perhaps non-obvious, acoustic cues related to sincerity.
  • Spectrograms outperform mel-spectrograms: For the DEEP SPECTRUM features, spectrogram representations generally yielded better results than mel-spectrograms for sincerity detection, contrary to findings in some other emotion-related tasks.
  • Speaker bias is likely: The SICV partitioning schema, which accounts for speaker variability more robustly, consistently outperformed the C-SIF schema, suggesting that sincerity cues might have a speaker-specific bias.
  • Apologies are complex: The paper highlights that even in acted apologies, sincerity is a nuanced trait that is challenging to capture computationally. The results, while promising, are not perfect, underscoring the complexity of human emotion expression.

Key Results

  • Classification: The best performance for binary sincerity classification was achieved using the spectrogram-based features (DS-4) with the SICV schema, yielding an Unweighted Average Recall (UAR) of 79.2%. This is a significant improvement over the C-SIF schema, which achieved a maximum UAR of 70.2% with EGEMAPS features.
  • Regression: For predicting the degree of sincerity, the spectrogram-based features (DS-2 and DS-4) also performed best under the SICV schema, achieving a Root Mean Square Error (RMSE) of 0.395 (normalized range [-1.51, 1.72]). This indicates a relatively good prediction of the continuous sincerity score.
  • Competitiveness of traditional features: While deep features generally excelled, traditional features like EGEMAPS and COMPARE remained competitive, particularly in the C-SIF schema for classification, showing that they still hold value.

Limitations and Future Work

  • Acted vs. "In-the-wild" speech: The SINA-C corpus consists of acted apologies. The authors acknowledge that this may not fully represent the nuances of spontaneous, "in-the-wild" apologies, which could be more challenging to detect sincerity from.
  • Original COMPARE baseline: The paper notes that their regression results did not beat the original COMPARE 2016 challenge baseline (0.609 p). They speculate this might be due to differences in the development strategy (not using leave-one-speaker-out cross-validation) and optimizing for RMSE instead of Spearman's correlation coefficient (p).
  • Future Directions:
    • Investigate speaker bias further: Explore how gender, age, and nativeness influence sincerity detection.
  • Transfer learning: Leverage knowledge from larger datasets related to similar traits like likability, trustworthiness, and deception.
  • Data augmentation: Apply techniques from computer vision to augment the sincerity dataset.
  • "In-the-wild" data: Crawl and analyze YouTube data related to sincerity and authenticity for more realistic scenarios.
  • Prosody and lexical content: Further analyze the interplay of prosodic features and the actual words used in apologies.

Summary updated

BibTeX
@inproceedings{baird2019sincerity,
  author = {Alice Baird and Eduardo Coutinho and Julia Hirschberg and Björn W. Schuller},
  title = {{Sincerity in Acted Speech: Presenting the Sincere Apology Corpus and Results}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure