Unveiling the acoustic properties that describe the valence dimension

Carlos Busso, Tauhidur RahmanInterspeech 2012

TL;DR

This paper tackles the challenge of recognizing emotions in speech based on their "valence" (positive vs. negative), a dimension notoriously difficult to capture with acoustic features compared to arousal. By systematically analyzing a wide range of acoustic features and employing a novel controlled experiment, the authors identify spectral and fundamental frequency (F0) features as most promising for valence discrimination.

What Problem Does This Paper Address?

Recognizing human emotions from speech is crucial for building more intuitive human-computer interfaces. While speech analysis has made strides in identifying emotions based on arousal (e.g., excited vs. calm), distinguishing between emotions with similar arousal but different valence (e.g., happy vs. angry) remains a significant hurdle. This difficulty has practical implications, particularly in understanding conditions like depression and PTSD, where valence is a key indicator. Existing research shows that common acoustic features that work well for arousal often fail to differentiate valence, leading to confusion between seemingly similar emotional states. This paper aims to pinpoint the specific acoustic properties that do effectively capture this valence dimension.

Methods

The authors employ a two-pronged approach. First, they conduct a regression analysis using Support Vector Regression (SVR) to predict valence from various groups of acoustic features. They train separate SVR models for energy, F0, voice quality, spectral, MFCC, and RASTA features, and then use correlation feature selection (CFS) to identify the most informative subsets.

Second, to isolate valence-specific features, they introduce a novel controlled experiment. This involves selecting speech samples that are perceived as having similar activation levels but distinctly different valence (positive vs. negative). They then use logistic regression to analyze individual features within these carefully chosen clusters, employing a log-likelihood ratio test to statistically assess the discriminative power of each feature.

Datasets

The study utilizes the Vera am Mittag (VAM) database. This corpus consists of 12 hours of audiovisual recordings from 47 speakers, featuring dialogues from a German TV show where guests discuss personal problems. The utterances are annotated by 17 raters for continuous attributes of activation, valence, and dominance, using a self-assessment manikin (SAM) scale, with average values mapped to the range [-1, 1] as ground truth. The paper uses a four-fold speaker-independent cross-validation scheme.

Key Insights and Ideas

  • Valence is Tricky: The paper confirms previous findings that acoustic features are significantly less effective at predicting valence compared to activation and dominance.
  • Spectral and F0 Reign Supreme: Spectral features and fundamental frequency (F0) emerge as the most informative feature groups for predicting valence.
  • Controlled Experiment is Key: The novel approach of selecting samples with similar activation but different valence is crucial for isolating features that specifically capture valence.
  • F0 Distribution Matters: Differences in F0 distribution, such as positive skewness and longer durations of low F0 values, are linked to positive valence.
  • Spectral Envelope is Informative: RASTA features, which capture the spectral envelope and reduce noise effects, are highly discriminative for valence. Specific filter bank coefficients within the 900-1300Hz range are particularly important.
  • Rising Spectral Roll-off: For positive valence, there's a tendency for the spectral roll-off frequency to increase from frame to frame, suggesting rising spectral energy.

Key Results

  • After feature selection using CFS, SVR models achieved a correlation of 0.3245 for valence prediction, significantly lower than for activation (0.8035) and dominance (0.7637).
  • When analyzing feature groups individually, spectral and F0 features yielded the best correlation for valence prediction (around 0.27), outperforming energy and voice quality features.
  • The controlled logistic regression analysis identified 435 relevant features out of 4,368 that significantly discriminated between positive and negative valence.
  • Spectral features (including RASTA and MFCC) accounted for over 80% of these relevant features, with energy and F0 features making up only about 10%.
  • The top-ranked feature was F0dltime25 (duration when F0 is below its 25% range), which was found to be higher in positive valence sentences.
  • RASTA features were the best performing group, with specific coefficients (e.g., Rfilt[9]prctl1.0, Rfilt[10]prctl1.0, Rfilt[11]prctl1.0) being highly discriminative.

Limitations and Future Work

  • Limited Activation Range: The current analysis primarily focused on samples with neutral activation. The authors plan to extend the study to samples with similar high/low activation but different valence.
  • Database Limitations: The authors note that current emotional databases may not fully cover the entire activation-valence space, necessitating the development of new, emotionally balanced corpora.
  • Articulatory Variables: To gain a deeper understanding of valence modulation in expressive speech, the authors plan to investigate articulatory variables, which could provide insights into the physical mechanisms behind valence changes. This could guide the design of more robust emotion recognition systems.

Summary updated

BibTeX
@inproceedings{busso2012unveiling,
  author = {Carlos Busso and Tauhidur Rahman},
  title = {{Unveiling the acoustic properties that describe the valence dimension}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure