Speaker Sincerity Detection based on Covariance Feature Vectors and Ensemble Methods

Mohammed Senoussaoui, Patrick Cardinal, Najim Dehak, Alessandro Lameiras KoericharXiv 2019 · OpenReview

TL;DR

This paper introduces a novel approach to automatically detect speaker sincerity by using covariance-based features derived from speech signals. These features, combined with ensemble regression models, show promising improvements over existing methods on the Sincerity Speech Corpus.

What Problem Does This Paper Address?

Accurately gauging a speaker's sincerity is a complex task for humans, relying on subtle vocal cues beyond the literal meaning of words. This challenge is particularly relevant in computational paralinguistics, a field aiming to equip machines with human-like emotional understanding. The paper tackles the problem of automatically estimating the degree of perceived sincerity in speech, a task that was a specific challenge in the ComParE 2016 competition. The goal is to move beyond traditional speech analysis to capture the nuances of sincerity in vocal delivery.

Methods

The core of this research lies in proposing and evaluating novel feature extraction techniques for sincerity detection. The authors introduce covariance-based features (CF), which capture the relationships between different short-term acoustic features (like MFCCs and their derivatives). Instead of just looking at individual feature statistics (mean, variance), CFs compute pairwise statistics, effectively modeling how features co-vary. This is visualized as creating a covariance matrix from short-term features and then vectorizing its upper or lower triangular part.

To further enhance these features, they also propose augmented covariance features (ACF), which combine CFs with individual statistics like mean, skewness, and kurtosis. These new features are then used with established machine learning algorithms: Support Vector Regression (SVR), Relevance Vector Machines (RVM), and Kernelized Ridge Regression (KRR). The paper also explores combining these novel features with the established ComParE acoustic feature set and investigates ensemble methods (feature-level and score-level fusion) to leverage potential complementarities.

Datasets

The primary dataset used for evaluation is the Sincerity Speech Corpus. This corpus contains speech samples from 32 speakers, with each instance rated on a sincerity scale from 0 (not sincere at all) to 4 (extremely sincere) by at least 13 annotators. The dataset was split into development and test sets. For training their i-vector extractor (a component not directly used for sincerity detection but mentioned as a baseline comparison), they utilized the TIMIT dataset, which consists of recordings from 630 speakers.

Key Insights and Ideas

  • Beyond individual statistics: The paper highlights that modeling the relationships between acoustic features (via covariance) can be more informative for sincerity detection than just looking at individual feature statistics. This suggests that the interplay of vocal characteristics is crucial for conveying sincerity.
  • Complementarity of features: Combining the proposed covariance-based features with existing acoustic features (like ComParE) and using ensemble methods leads to performance improvements, indicating that different feature sets capture different aspects of sincerity.
  • Data efficiency of CF: Covariance features are noted for their ease of extraction and lack of need for extensive training data, making them suitable for scenarios with limited data, unlike i-vectors.

Key Results

On the development set of the Sincerity Speech Corpus, the proposed covariance-based features, when used with SVR, achieved an 8.1% relative improvement in Spearman's correlation coefficient over the baseline system. Further improvements were observed when combining features and using ensemble methods, with the best result reaching a Spearman's correlation of 0.562 on the development set. However, the performance on the test set was not as strong, with the best result being 0.547, which was lower than the baseline's reported 0.602. The authors attribute this discrepancy to the very high dimensionality of the CF features.

Limitations and Future Work

A significant limitation identified is the high dimensionality of the covariance features, which negatively impacted performance on the test set. The authors explored dimensionality reduction using Principal Component Analysis (PCA) on the input space, showing some improvement but still not matching the baseline on the test set.

Future work suggested includes:

  • More sophisticated dimensionality reduction: Investigating alternative methods to reduce the dimensionality of CF features.
  • Dynamic feature selection: Exploring mechanisms to select the best feature set for each individual speaker, as an "oracle" approach showed potential for much higher performance (0.740 correlation).
  • Analyzing instance similarity: Identifying similarities between test instances and speakers in the development set to inform dynamic selection strategies.

Summary updated

Abstract

Automatic measuring of speaker sincerity degree is a novel research problem in computational paralinguistics. This paper proposes covariance-based feature vectors to model speech and ensembles of support vector regressors to estimate the degree of sincerity of a speaker. The elements of each covariance vector are pairwise statistics between the short-term feature components. These features are used alone as well as in combination with the ComParE acoustic feature set. The experimental results on the development set of the Sincerity Speech Corpus using a cross-validation procedure have shown an 8.1% relative improvement in the Spearman's correlation coefficient over the baseline system.
BibTeX
@misc{senoussaoui2019speaker,
  author = {Mohammed Senoussaoui and Patrick Cardinal and Najim Dehak and Alessandro Lameiras Koerich},
  title = {{Speaker Sincerity Detection based on Covariance Feature Vectors and Ensemble Methods}},
  year = {2019},
  eprint = {1904.11641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.11641},
}
Figure