Prompting Audios Using Acoustic Properties For Emotion Representation

Hira Dhamyal, Benjamin Elizalde, Soham Deshmukh, Huaming Wang, Bhiksha Raj, Rita Singh

Abstract

Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.

Introduction

Traditional Speech Emotion Recognition (SER) systems often rely on categorical labels like "happy," "sad," or "angry." While convenient, this approach fails to capture the nuanced continuum of human emotion. For instance, "anger" can manifest as a quiet, low-pitched hiss or a loud, high-pitched shout. To address this, researchers from CMU and Microsoft propose the use of acoustic prompts—natural language descriptions that incorporate physical properties of sound to provide a richer emotional context.

Methodology

The proposed framework leverages a Contrastive Language-Audio Pretraining (CLAP) architecture. This setup consists of an audio encoder (CNN14) and a text encoder (BERT) designed to map audio signals and their corresponding textual descriptions into a shared embedding space.

The CLAP architecture for training on text-audio pairs and testing on emotion classes.

The core innovation lies in the automatic generation of prompts based on four acoustic correlates:

  1. Pitch: Binned into categories based on sex-specific frequency ranges.
  2. Intensity: Categorized into high or low levels.
  3. Speech Rate: Measuring the speed of delivery.
  4. Articulation Rate: Focusing on the clarity and speed of phonetic segments.

By combining these features with standard emotion labels, the system generates descriptive prompts such as "high female pitch anger." During training, Prompt Augmentation (PA) is used, where a single audio clip is paired with multiple independent acoustic prompts, forcing the model to learn the underlying relationships between physical acoustic features and emotional semantics.

Experimental Setup

The model was trained and evaluated using a diverse set of six datasets, including IEMOCAP, MELD, CREMA-D, and RAVDESS, alongside a "4D" backbone collection of audio captioning data (ClothoV2, AudioCaps, etc.). Performance was measured across two primary tasks: Speech Emotion Recognition (SER) and a newly introduced task, Emotion Audio Retrieval (EAR).

Results

Speech Emotion Recognition (SER)

The inclusion of acoustic prompts led to a notable performance boost. On the RAVDESS dataset, the model achieved a 3.8% relative accuracy improvement over the baseline. Specifically, in a fine-tuning scenario, accuracy increased from 68.69% to 72.46%. Among the individual acoustic properties, pitch was identified as the most influential factor in improving recognition accuracy.

Emotion Audio Retrieval (EAR)

The EAR task evaluates the model's ability to retrieve relevant audio clips based on complex text queries (e.g., "Find me clips of high-intensity fear"). The model trained with Prompt Augmentation (PA) consistently outperformed baselines across all Precision@K metrics.

Precision@K metrics for Emotion Audio Retrieval across different query types.

As shown in the results, for class label queries, P@1 increased from 0.50 to 0.75 when using the full acoustic prompting suite. The model successfully learned to associate specific acoustic descriptors with their corresponding audio signals, even in the absence of explicit emotion labels.

Discussion and Future Work

While acoustic prompting significantly enhances emotion representation, some limitations remain. The current approach uses coarse binning (2-4 categories) and fixed thresholds from literature. Future research could explore more fine-grained or continuous representations and data-centric binning to adapt to the specific distributions of different datasets. Additionally, incorporating more complex features like jitter, shimmer, or conversational context could further refine the model's understanding of emotional nuance.

Summary updated

Abstract

Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.
BibTeX
@inproceedings{dhamyal2024prompting,
  author = {Hira Dhamyal and Benjamin Elizalde and Soham Deshmukh and Huaming Wang and Bhiksha Raj and Rita Singh},
  title = {{Prompting Audios Using Acoustic Properties For Emotion Representation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MBrp1qS0F5},
}
Figure