Multi-system fusion of extended context prosodic and cepstral features for paralinguistic speaker trait classification

Michelle Hewlett Sanchez, Aaron Lawson, Dimitra Vergyri, Harry BrattInterspeech 2012

TL;DR

This paper tackles the challenge of classifying subtle "beyond-the-words" speaker traits like likeability and personality from speech. By fusing multiple acoustic feature sets (prosodic, cepstral, and spectral) and various classification models, the authors achieve competitive results, outperforming the challenge baseline in most categories. This highlights the power of multi-modal feature fusion for nuanced speaker analysis.

What Problem Does This Paper Address?

While automatic speech recognition (ASR) has made strides in understanding what is being said, there's a growing interest in what speech reveals about the speaker. This includes identifying characteristics beyond basic identity, such as age, dialect, emotion, and even personality traits. This paper focuses on classifying seven specific paralinguistic speaker traits: likeability, intelligibility, openness, conscientiousness, extraversion, agreeableness, and neuroticism. These traits are subtle and not explicitly encoded in the linguistic content, making their detection a challenging problem. The authors aim to leverage a combination of acoustic features and machine learning models to accurately predict these traits.

Methods

The core of this paper lies in its multi-system fusion approach. They extract a rich set of features, combining:

  • Prosodic Features: These capture the rhythm, intonation, and energy of speech. They include fundamental frequency (pitch) and energy, both in their raw form and normalized (FONorm, FOZNorm, EnergyOverVoiced). They also explore prosodic polynomial coefficients, which model the dynamics of speech over small, overlapping windows, capturing subtle variations.
  • Spectral Features: Standard Mel-Frequency Cepstral Coefficients (MFCCs) are used, including their derivatives.
  • Shifted Delta Cepstrum (SDC): This technique enhances MFCCs by incorporating temporal context, providing a more robust representation.

These features are then fed into several classification systems:

  • Support Vector Machines (SVM): A standard and powerful classifier, used with both raw and reduced feature sets.
  • Gaussian Mixture Models (GMM) with Universal Background Models (UBM): This approach, common in speaker identification, involves training a UBM on general speech data and then adapting it to specific classes (speaker traits). They employ two variations:
  • MAP Adaptation: Maximum A Posteriori estimation to adapt the UBM means.
  • Nuisance Compensation (NC): A more advanced technique to reduce unwanted variability (like channel effects or speaker-specific characteristics) to better highlight trait-specific differences.
  • Eigenchannel Systems: Used with SDC features, these systems aim to compensate for nuisance factors.
  • L2 and SVM Classifiers with Feature Reduction: Given the large initial feature set (6125 dimensions), they employ a feature selection method based on individual feature efficacy (maximum average recall) to select the most relevant features for each task.

The key innovation is the fusion of scores from these diverse systems. They use logistic regression to calibrate and combine the outputs of multiple classifiers, aiming to leverage the complementary strengths of each system.

Datasets

The paper uses data from the Interspeech Speaker Trait Challenge. While specific details about the challenge dataset (e.g., size, number of speakers, recording conditions) are not extensively detailed beyond referencing [12], it's noted that the available data for paralinguistic traits is limited. To address this, they mention using other available English data, labeled only with speaker information, to train some model parameters, particularly for the UBMs.

Key Insights and Ideas

  • Fusion is Key: The most significant finding is that combining multiple systems and feature sets dramatically improves performance. The fused system consistently outperforms individual systems, demonstrating that no single approach captures all the necessary information for these subtle traits.
  • Prosodic Dynamics Matter: The use of prosodic polynomial coefficients, which capture the subtle temporal variations in pitch and energy, proves valuable, especially when combined with other features.
  • Feature Reduction is Necessary: With a very large initial feature set, effective feature selection is crucial for achieving good performance and computational efficiency. Their efficacy-based feature selection method shows promise.
  • UBM-GMM Adaptability: The UBM-GMM framework, particularly with nuisance compensation, is shown to be effective for paralinguistic trait classification, adapting well to the specific challenges.

Key Results

The paper reports strong results, particularly with their fused system.

  • Outperforming Baselines: The fused system achieved the best performance on the development data, outperforming the challenge baseline by a significant margin: 7.5% for likeability and 6% for intelligibility. For personality traits, the fusion system outperformed the baseline by 9% for openness, 5% for extraversion, and 3% for neuroticism.
  • Test Data Performance: On the test data, the fused system was the only submission. It outperformed the challenge baseline in openness and performed competitively in intelligibility.
  • Individual System Performance: While fusion was the star, individual systems like the Eigenchannel MFCC SDC system showed strong performance on specific traits (e.g., 64.6% for likeability).

Limitations and Future Work

  • Data Mismatch: The authors note a potential mismatch between the development and test data, as a single system that performed well on development data did not perform as well on the test set. This suggests that generalization across different data splits can be a challenge.
  • Limited Data: The scarcity of labeled paralinguistic data is a recurring theme, prompting the use of auxiliary speaker-labeled data for UBM training.
  • Future Directions: While not explicitly detailed, the success of fusion suggests that exploring even more diverse feature sets and advanced fusion techniques could lead to further improvements. Investigating the specific reasons for the development-test data mismatch would also be a valuable future direction.

Summary updated

BibTeX
@inproceedings{sanchez2012multi,
  author = {Michelle Hewlett Sanchez and Aaron Lawson and Dimitra Vergyri and Harry Bratt},
  title = {{Multi-system fusion of extended context prosodic and cepstral features for paralinguistic speaker trait classification}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure