Prompting Audios Using Acoustic Properties for Emotion Representation

Hira Dhamyal, Benjamin Elizalde, Soham Deshmukh, Huaming Wang, Bhiksha Raj, Rita Singh

TL;DR

Researchers from CMU and Microsoft have moved beyond simple "happy" or "angry" labels for speech emotion by using acoustic prompts. By automatically generating natural language descriptions based on physical properties like pitch and intensity (e.g., "high-intensity anger"), they trained a contrastive learning model that significantly improves Speech Emotion Recognition (SER) and enables a new task: Emotion Audio Retrieval (EAR).

What Problem Does This Paper Address?

Current AI models usually treat emotions as discrete categories (the "Plutchik wheel" or "Ekman model"). However, emotion exists on a continuum. Labeling two different audio clips as "anger" ignores the nuance between a cold, quiet seething and a loud, high-pitched scream.

This categorical approach limits a model's ability to capture the diversity of emotional expression. The authors argue that natural language is a better medium for capturing these fine-grained characteristics. The challenge is: how do we automatically generate these descriptive "prompts" at scale without manual labeling?

Methods

The authors utilize a Contrastive Language-Audio Pretraining (CLAP) architecture. This involves an audio encoder (CNN14) and a text encoder (BERT) that are trained to map audio and their corresponding text descriptions into a shared multidimensional space.

The core innovation is Acoustic Prompting:

  1. Feature Extraction: They extract four low-level acoustic correlates of emotion: Pitch, Intensity, Speech Rate, and Articulation Rate.
  2. Binning: These values are binned into categories (e.g., "high" vs "low") based on literature-standard thresholds. For example, pitch is binned into four categories based on sex-specific frequency ranges.
  3. Prompt Generation: These features are combined with the emotion label to create prompts like "high female pitch anger" or "low intensity sadness."
  4. Prompt Augmentation: Instead of one label per clip, they pair a single audio clip with multiple independent acoustic prompts during training, forcing the model to learn the relationship between specific physical sounds and emotional semantics.

Datasets

The study uses a robust collection of six publicly available emotion datasets:

  • CMU-MOSEI & CMU-MOSI: Multimodal sentiment and emotion.
  • IEMOCAP: Dyadic motion capture (widely used in SER).
  • MELD: Multi-party Easter-egg dialogue.
  • CREMA-D: Crowd-sourced emotional multimodal actors.
  • RAVDESS: Emotional speech and song.

For the backbone pre-training, they use the "4D" collection, which includes audio captioning datasets like ClothoV2, AudioCaps, MACS, and FSD50K.

Key Insights and Ideas

  • Acoustics as a Bridge: By grounding text prompts in physical audio properties (like pitch), the model learns a more "honest" representation of emotion that isn't just memorizing class labels.
  • Pitch is King: Among the four acoustic properties tested, pitch was found to be the most effective single prompt for improving emotion recognition accuracy.
  • The Power of Retrieval: The authors introduce Emotion Audio Retrieval (EAR). This allows a user to search a database using a text query (e.g., "Find me clips of high-intensity fear"). This has massive implications for speech forensics, search engines, and social media monitoring.
  • Zero-Shot vs. Fine-Tuning: While acoustic prompts helped significantly in fine-tuning, they struggled in "leave-one-out" zero-shot scenarios due to distribution shifts between different datasets (e.g., "high intensity" in one dataset might be "medium" in another).

Key Results

  • Speech Emotion Recognition (SER): On the RAVDESS dataset, the acoustic prompting method achieved a 3.8% relative accuracy improvement over the baseline. In a fine-tuning setup, they saw an absolute improvement of 3.77% (moving from 68.69% to 72.46%).
  • Emotion Audio Retrieval (EAR): The model trained with Prompt Augmentation (PA) consistently outperformed the baseline across all Precision@K metrics. For class label queries, P@1 jumped from 0.50 to 0.75.
  • Acoustic Association: The model successfully learned to associate specific sounds with text; it could accurately retrieve "high pitch" audio when queried with that specific text, even without being explicitly told the emotion.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Limited Features: They only used four acoustic properties. Future work could include more complex features like jitter, shimmer, or spectral flux.
  • Coarse Binning: The features were binned into only 2 or 4 categories. More fine-grained or continuous representations might capture even more nuance.
  • Thresholding: Currently, they use fixed thresholds from literature. They suggest moving toward data-centric binning, where thresholds are determined dynamically based on the specific dataset's distribution.
  • Context: The current prompts focus on how something is said, but not who is saying it or the context of the conversation.

Summary updated

Abstract

Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. ‘acoustic prompts’. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model’s performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.
BibTeX
@inproceedings{dhamyal2024prompting,
  author = {Hira Dhamyal and Benjamin Elizalde and Soham Deshmukh and Huaming Wang and Bhiksha Raj and Rita Singh},
  title = {{Prompting Audios Using Acoustic Properties for Emotion Representation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MBrp1qS0F5},
}

Loading related content...

Figure