Predicting Emotion Perception Across Domains: A Study of Singing and Speaking

Biqiao Zhang, Emily Mower Provost, Robert Swedberg, Georg EsslAAAI 2015 · DOI

Humans possess a remarkable ability to recognize emotions across various vocal expressions, whether in casual conversation or melodic song. This suggests a shared underlying mechanism for affective communication. However, most automated Emotion Recognition (ER) systems are specialized for a single domain. This paper investigates the feasibility of cross-domain ER, specifically looking at how emotion perception generalizes between speaking and singing.

Modeling Subjective Emotion

A core challenge in ER is the inherent subjectivity of human perception. Rather than using discrete labels like "Happy" or "Sad," the researchers utilized the Valence-Activation (V-A) dimensional space. To capture the variability among evaluators, they employed Kernel Density Estimation (KDE). Instead of a single point estimate, KDE models emotion as a probability distribution over the V-A space, reflecting the consensus and divergence of human ratings.

Modeling emotion perception as a probability distribution using KDE: (a) raw ratings, (b) continuous density, and (c) binned density.

Methodology: Bridging the Domain Gap

The transition from speaking to singing introduces significant acoustic shifts due to musical constraints like pitch and rhythm. To identify features that remain stable across these domains, the authors developed a novel feature selection method. They adapted the minimum Redundancy Maximum Relevance (mRMR) algorithm by replacing the standard Pearson correlation with a Cross-Domain Correlation Coefficient (CDCC). This allowed the selection of "universal" features that are both relevant to emotion and consistent across speech and song.

The Power of Visual Cues

One of the study's most significant findings is the role of visual information. While acoustic features change drastically when a person starts singing, facial expressions remain relatively consistent. The researchers used facial action units and saliency maps to capture these visual cues.

Visual features and saliency maps used to capture emotional expression across domains.

The results demonstrated that visual features are far superior to acoustic features for cross-domain valence prediction. In many cases, audio-only models failed completely ($R^2 \approx 0$) when moving from one domain to another, while video-only models maintained significant predictive power (up to 0.44 $R^2$).

Key Insights

  • Activation vs. Valence: Emotional "activation" (intensity) generalizes well across domains, suggesting that the energy of an emotion is expressed similarly in speech and song. In contrast, "valence" (positivity/negativity) appears more domain-specific and harder to predict using acoustic data alone.
  • Visual Dominance in Singing: The study suggests that when perceiving emotion in singing, both humans and models rely more heavily on visual cues for valence than they do for speaking.
  • Cross-Domain Features (CDF): Models using the proposed CDFs provided a robust middle ground, outperforming domain-specific models in cross-domain tasks while maintaining high accuracy within domains.

Conclusion

By moving beyond siloed ER systems, this research highlights the importance of multi-modal, cross-domain approaches. The findings underscore that while our voices may change to meet musical demands, our faces tell a consistent emotional story, providing a vital bridge for understanding human affect across different forms of vocal expression.

Summary updated

BibTeX
@inproceedings{zhang2015predicting,
  author = {Biqiao Zhang and Emily Mower Provost and Robert Swedberg and Georg Essl},
  title = {{Predicting Emotion Perception Across Domains: A Study of Singing and Speaking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9334},
}
Figure