Abstract: Studies on emotion recognition (ER) show that combining lexical and acoustic information results in more robust and accurate models. The majority of the studies focus on settings where both modalities are available in training and evaluation. However, in practice, this is not always the case; getting ASR output may represent a bottleneck in a deployment pipeline due to computational complexity or privacy-related constraints. To address this challenge, we study the problem of efficiently combining acoustic and lexical modalities during training while still providing a deployable acoustic model that does not require lexical inputs. We first experiment with multimodal models and two attention mechanisms to assess the extent of the benefits that lexical information can provide. Then, we frame the task as a multi-view learning problem to induce semantic information from a multimodal model into our acoustic-only network using a contrastive loss function. Our multimodal model outperforms the previous state of the art on the USC-IEMOCAP dataset reported on lexical and acoustic information. Additionally, our multi-view-trained acoustic network significantly surpasses models that have been exclusively trained with acoustic features.
Introduction
Emotion recognition (ER) is inherently multimodal; humans express feelings through both what they say (lexical content) and how they say it (acoustic prosody). While models utilizing both modalities consistently perform best, they face a significant hurdle in real-world production: the dependency on Automatic Speech Recognition (ASR). Generating high-quality transcripts in real-time is computationally expensive, adds latency, and can raise privacy concerns.
This research addresses a critical deployment challenge: How can we build an acoustic model that benefits from the rich semantic information found in text, even when that text is unavailable during evaluation? The authors propose a hierarchical multimodal model that aligns information at the word level and then uses this model as a "teacher" to train an acoustic-only "student" via multi-view learning.
Methodology
1. The Hierarchical Multimodal Model
Rather than simple feature concatenation at the utterance level, the proposed architecture aligns audio and text at the word level using "acoustic words."
- Acoustic Words: Frame-level audio features are processed through a BLSTM for each word segment, creating a vector representation of how a specific word sounds.
- Gated Multimodal Unit (GMU): This acts as a modality-based attention mechanism. At each word step, the GMU dynamically decides which modality is more informative. For instance, in sarcastic or filler-heavy speech, the model may prioritize acoustic tone over literal text.
- Context-based Attention: A final attention layer identifies which words within the entire utterance are most critical for determining the final emotion label.
2. Multi-view Learning (Teacher-Student Setup)
To improve the acoustic-only model for deployment, the researchers treat the multimodal network as a "teacher" to guide an acoustic-only "student."
- Contrastive Loss: During training, the model forces the acoustic-only network to produce hidden representations ($h_a$) that are close to the multimodal representations ($h_m$) for the same utterance, while pushing them away from representations of different emotions.
- Knowledge Distillation: The authors utilize Kullback-Leibler (KL) divergence to encourage the acoustic-only student to match the probability distribution (the "soft labels") produced by the multimodal teacher.
Experimental Setup and Results
The models were evaluated on the USC-IEMOCAP dataset, a standard benchmark containing ~12 hours of audiovisual data across four emotional categories: anger, happiness, neutral, and sadness. Audio features were extracted using the OpenSMILE ComParE 2013 set, while lexical features utilized contextualized ELMo embeddings.
Key Findings
- New State-of-the-Art: The hierarchical multimodal model achieved 71.69% Unweighted Accuracy (UA), surpassing the previous state-of-the-art benchmark of 70.79%.
- Semantic Induction: The most significant takeaway is the boost to acoustic-only performance. A standard acoustic model achieved 54.43% UA. When trained using the multi-view teacher-student approach, performance jumped to 59.76% UA—a substantial ~5.3% absolute improvement without adding any computational complexity at inference time.
- Dynamic Weighting: Visualization of GMU weights confirmed the model relies more on acoustics for filler words and more on lexical content for high-sentiment words like "wonderful" or "stupid."
Conclusion
By framing emotion recognition as a multi-view learning problem, this work demonstrates that acoustic models can be "taught" to recognize the sound of semantic importance. While the approach requires precise word-level alignment during training, it offers a powerful path forward for deploying high-performance emotion recognition systems in environments where ASR is not feasible.
Summary updated