TL;DR
This paper introduces N-CORE, a novel framework for disentangling emotion and speaker information from nonverbal vocalizations (NVVs) like laughter and sighs. By using multiple perturbed views of audio samples, N-CORE achieves state-of-the-art performance in classifying emotions and speakers from NVVs, addressing a critical gap in understanding human communication.
What Problem Does This Paper Address?
Nonverbal vocalizations (NVVs) are crucial for human communication, conveying rich emotional and social information without words. However, computationally analyzing NVVs is challenging due to a lack of clear linguistic anchors (like words in speech) and often imbalanced, biased datasets. This makes it difficult to isolate specific features like emotion from speaker characteristics, leading to models that confuse the two. Existing methods for disentangled representation learning (DRL), which aim to separate these features, haven't been effectively applied to NVVs because NVVs lack the structured linguistic cues that traditional DRL methods rely on. This paper tackles the problem of learning disentangled representations for NVVs to improve their analysis for applications ranging from AI assistants to clinical diagnostics.
Methods
The core of this paper is the N-CORE (N-View COnsistency REgularization) framework. N-CORE is designed to be backbone-agnostic, meaning it can work with various pre-trained audio foundation models (like HuBERT or Wav2Vec2). The key innovations are:
- N-View Consistency Regularization: Instead of using just one perturbed version of an audio sample, N-CORE generates N different perturbed views. This exposes the model to a wider range of variations, forcing it to learn representations that are invariant to irrelevant features (like speaker identity when classifying emotion, or vice-versa) and more robust to unseen distortions. A pairwise distance loss is used to ensure that all views of a sample map to a similar representation in the latent space.
- Feature-Invariant Audio Perturbation: The authors introduce novel perturbation functions:
- Emotion Perturbation (
pe): This function aims to disrupt emotional cues while preserving speaker characteristics. It involves randomly permuting frequency bands in the spectrogram, normalizing intensity, and flattening the pitch contour. - Speaker Perturbation (
ps): This function, adapted from prior work, aims to disrupt speaker characteristics while preserving content. It involves scaling formant frequencies and pitch, and applying a random equalizer. - Dual Classification Heads with GRL: N-CORE uses two classification heads: a primary head for the target task (e.g., emotion classification) and a secondary adversarial head with a Gradient Reversal Layer (GRL). The GRL encourages the model to learn representations that are invariant to the secondary, irrelevant feature (e.g., speaker identity).
Datasets
The authors evaluate their framework on three nonverbal vocalization datasets:
- VIVAE: A corpus of acted non-speech emotion vocalizations from 11 female actors.
- ReCANVO: A dataset of spontaneous NVVs from individuals with neurodevelopmental disorders, including autism.
- ReCANVO-Balanced: A balanced subset derived from ReCANVO to mitigate class imbalance issues.
Key Insights and Ideas
- NVVs are fundamentally different: The paper highlights that NVVs lack the phoneme-based priors that make DRL easier for spoken language. This necessitates new approaches to perturbation and regularization.
- Multi-view perturbation is key: Using multiple perturbed views (N > 1) significantly improves robustness and performance compared to single-view methods, especially for NVVs where lexical anchors are absent.
- Emotion and speaker are deeply intertwined in NVVs: The authors show that disentangling these features is challenging but achievable, and crucial for accurate classification.
- Domain-specific pre-training isn't always enough: While models pre-trained on NVVs (like Voc2Vec) show promise, general-purpose foundation models like HuBERT, when combined with N-CORE, can achieve superior or comparable results, suggesting the power of advanced representation learning.
- DRL excels in specific scenarios: Disentanglement proves particularly effective in datasets with homogeneous speaker demographics (like VIVAE) for speaker classification, suggesting it helps overcome subtle variations.
Key Results
- State-of-the-art performance: N-CORE (both N-COREWav2Vec2 and N-COREHUBERT) outperforms all other tested methods on the VIVAE and ReCANVo-Balanced datasets for both emotion and speaker classification tasks.
- Robustness to imbalance: N-CORE shows competitive performance on the imbalanced ReCANVO dataset, and its balanced version (ReCANVo-Balanced) further highlights its effectiveness.
- Perturbation validation: The proposed emotion perturbation function (
pe) and the adapted speaker perturbation function (ps) were validated, showing that applying the correct perturbation pairing leads to improved classification performance, confirming their efficacy in disrupting specific features. - Optimal N: The study found that using N=5 perturbed views generally yields the best results on VIVAE, suggesting a sweet spot for balancing diversity and avoiding overfitting.
Limitations and Future Work
- Limited disentanglement scope: The current study focuses on disentangling emotion and speaker features. NVVs convey a much richer spectrum of paralinguistic information (intensity, intent, subtle cues) that N-CORE does not explicitly disentangle.
- Dataset constraints: The findings are based on a limited number of datasets, which may not capture the full variability of NVVs across different cultures, ages, acoustic environments, or clinical populations.
- Performance on imbalanced data: N-CORE's performance can be impacted by extreme data imbalances where speaker and affective cues are highly convoluted.
- Future Directions: The authors suggest exploring applications like privacy-preserving NVV encoding, disentangled voice conversion for NVVs, and clinical analysis of vocalizations from non- and minimally-speaking individuals. The backbone-agnostic design of N-CORE allows it to benefit from future advancements in large foundation models.
Summary updated