Representing Affect in Conversational Speech Datasets

231 concepts223 papers2 proposals

Speech Emotion Recognition (SER) seeks to bridge the gap between human affective communication and machine understanding. The core challenge lies in the fact that emotion is not a monolithic signal; it is a complex interplay of acoustic prosody, lexical content, and social context. To capture this, researchers have moved from simple, isolated recordings to sophisticated dyadic corpora that model how emotions evolve during interaction.

The Foundation of Acted and Scripted Corpora

Early progress in SER relied heavily on acted datasets, where professional actors portray specific emotions. The USC Interactive Emotional Dyadic Motion Capture (IEMOCAP) database has become a cornerstone of this field. Unlike datasets of isolated utterances, IEMOCAP captures long-form interactions between pairs of actors, providing a rich source of audio-visual data. Lee et al. demonstrate the value of this temporal depth by using bidirectional LSTMs to capture long-range context, treating emotional labels as random variables to account for the inherent uncertainty in how humans perceive affect.

Acted datasets like EMO-DB (Berlin) and CREMA-D also provide high-quality, labeled samples that allow researchers to isolate specific acoustic features. For instance, Phukan et al. found that embeddings from speaker recognition models (x-vectors) are surprisingly effective for emotion tasks on CREMA-D, suggesting that the vocal characteristics identifying a speaker are closely linked to those conveying their emotional state.

From Controlled Portrayals to Spontaneous Interaction

While acted speech provides clear signals, it often lacks the nuance of "real-life" emotion. This has led to a shift toward spontaneous datasets. Devillers et al. introduced the CEMO Corpus, using medical emergency call center dialogues to study emotions like relief and fear in high-stakes, authentic environments. Similarly, Arimoto et al. collected spontaneous speech from online gaming chats to estimate emotional intensity in natural, unscripted settings.

The transition to spontaneous data introduces technical hurdles, such as audio-visual misalignment. Carneiro et al. addressed this by refining the MELD dataset into MELD-FAIR, using active speaker localization to ensure that the emotional labels actually correspond to the person currently speaking. This refinement is crucial because, in real conversations, the "emotion" of a scene may be driven by a listener's reaction rather than the speaker's words.

The Multimodal and Contextual Shift

Modern SER recognizes that speech does not exist in a vacuum. The intuition is that emotion is multimodal and contextual.

  1. Multimodal Fusion: Kumar et al. proposed ParallelNet to fuse speech and image data, while Ringeval et al. showed that speech audio can even predict facial "Action Units" (muscle movements) when video is unavailable.
  2. Social Context: Emotions are dyadic. Zhang et al. introduced the Emotion Interaction and Transition (EIT) model, which argues that a speaker’s emotional state is heavily influenced by their interlocutor. By modeling this social "transition," they achieved higher accuracy on IEMOCAP than by looking at speakers in isolation.
  3. Linguistic Nuance: The meaning of words (lexical cues) often conflicts with the tone of voice (prosody). Dupuis et al. found that while younger adults can navigate these conflicts, older adults may struggle when lexical and prosodic cues disagree.

Addressing Subjectivity and Ambiguity

A recurring problem in this research area is that emotional labels are subjective. Two annotators might hear the same clip and disagree on whether the speaker is "frustrated" or "sad."

To handle this, Fujioka et al. developed meta-learning techniques to dynamically correct noisy labels during training. Other researchers focus on the type of emotion being measured. Busso et al. highlight the difficulty of distinguishing valence (positive vs. negative) compared to arousal (calm vs. excited), identifying specific spectral features that help isolate the valence dimension.

Furthermore, Ide et al. proposed a distinction between expressed emotion (what the speaker intends) and experienced emotion (what the listener feels). Their Japanese dialogue corpus demonstrates that predicting how a listener will feel is a significantly harder, yet more vital, task for creating empathetic AI. By moving toward these nuanced definitions of affect, the field continues to refine how machines interpret the complexities of human feeling.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_iemocap_dataset_19f1eeaa,
  title = {IEMOCAP dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iemocap-dataset/rdr_19f1eeaafe624374bdc07ae1e8f5566c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_iemocap_dataset_19f1eeaa_v1,
  title = {IEMOCAP dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iemocap-dataset/rdr_19f1eeaafe624374bdc07ae1e8f5566c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake