Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages

Heqing Zou, Fengmao Lv, Desheng Zheng, Eng Siong Chng, Deepu Rajan

Introduction

Speech Emotion Recognition (SER) remains a significant challenge for artificial intelligence, particularly when transitioning between different languages. While humans can often intuit emotional states regardless of the language spoken, most AI models are monolingual, failing to generalize across diverse linguistic rhythms and cultural expressions. This paper introduces a novel two-stage framework that leverages Large Language Models (LLMs) and contrastive learning to achieve state-of-the-art performance in multilingual emotion recognition, including "zero-shot" capabilities on languages entirely unseen during training.

Methodology: The Emotion Q-Former and Contrastive Learning

The proposed architecture aims to create speech representations that are simultaneously emotion-aware and language-agnostic.

System Architecture

The framework utilizes Whisper as a robust audio encoder to extract raw speech features. These features are passed through a specialized "bridge" called the Emotion Q-Former. This component uses learnable queries to distill emotion-specific information from the high-dimensional audio output, filtering out linguistic noise and background interference. Finally, LLaMA 3 serves as the reasoning engine, treating emotion recognition as a text-generation task where the model predicts emotional labels as words.

Two-Stage Training Pipeline

  1. Stage 1 (Alignment): The model is first trained on high-quality, human-annotated English datasets to align acoustic signals with emotional language.
  2. Stage 2 (Multilingual Refinement): The model is fine-tuned on the new M5SER dataset using a Language-agnostic Emotion-aware Contrastive (LEC) loss. This loss function forces the model to minimize the distance between identical emotions across different languages (e.g., "sad" in Mandarin vs. "sad" in French) while maximizing the distance between different emotional categories.

Data Resources: The M5SER Dataset

To address the scarcity of high-quality multilingual emotional data, the researchers developed the M5SER dataset. Using the SeamlessExpressive translator, they synthesized over 1,009 hours of emotional speech across five languages: French, German, Italian, Mandarin, and Spanish.

Distribution of emotions and languages within the M5SER dataset.

The model's robustness was further tested on true zero-shot benchmarks, including Greek (AESDD), Russian (RESD), and Persian (ShEMO), representing languages completely absent from the training data.

Key Results and Visual Analysis

The framework demonstrates superior performance on both traditional and zero-shot benchmarks. On the MELD dataset, the model achieved a Weighted Accuracy (WA) of 61.4%, surpassing previous state-of-the-art models like Vesper (53.5%). Most notably, the model reached a WA of 71.3% on the Persian ShEMO dataset in a zero-shot setting.

t-SNE visualizations of emotional audio distributions across the six primary languages, demonstrating clear clustering of emotional categories.

Visual analysis via t-SNE plots confirms the efficacy of the LEC loss. As shown in the visualizations, emotional categories form distinct, well-separated clusters across all languages, indicating that the model successfully learned to extract a universal emotional "core" while ignoring language-specific variations.

Conclusion and Future Directions

This research proves that combining LLMs with contrastive learning and high-quality synthetic data can effectively bridge the language gap in emotion recognition. While a performance gap still exists between zero-shot results and language-specific fine-tuning, the ability to recognize emotions in unseen languages like Persian and Greek marks a significant milestone. Future work will focus on reducing the computational cost of full fine-tuning and incorporating more "in-the-wild" authentic emotional data to complement synthetic training sets.

Summary updated

BibTeX
@inproceedings{zou2025large,
  author = {Heqing Zou and Fengmao Lv and Desheng Zheng and Eng Siong Chng and Deepu Rajan},
  title = {{Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages}},
  booktitle = {ICME},
  year = {2025},
  doi = {10.1109/icme59968.2025.11209040},
}
Figure