TL;DR
This paper introduces XLM-Muriel, a novel hard parameter-sharing architecture for multilingual, multi-label emotion detection. By leveraging a shared XLM-ROBERTa encoder with language-specific classification heads and a two-stage training strategy, the model achieves strong performance across 28 languages, including low-resource ones, demonstrating the effectiveness of this approach for bridging the gap in cross-lingual emotion understanding.
What Problem Does This Paper Address?
Understanding emotions expressed in text is a complex task, further complicated by the vast diversity of languages and the subtle ways emotions are conveyed. Traditional emotion recognition research often focuses on high-resource languages, leaving a significant gap in understanding and detecting emotions in many low-resource languages. Moreover, emotion detection is not a single problem but a collection of related challenges, including identifying the speaker's emotional state, discerning sentiment, and even gauging the emotional response triggered in a reader. This paper specifically tackles perceived emotion: inferring the emotion a majority would attribute to a speaker based on a text snippet, acknowledging that this can differ from the speaker's true feelings due to cultural context and individual differences. The challenge is to build a robust system that can handle multi-label classification (multiple emotions per text) across numerous languages simultaneously.
Methods
The core of XLM-Muriel is a hard parameter-sharing architecture built upon the XLM-ROBERTa-Large model. This means a single, powerful multilingual encoder is shared across all languages. To account for language-specific nuances in emotion expression, lightweight, language-specific classification heads are attached to this shared encoder.
The training strategy is a key innovation:
- Two-stage training:
- Stage 1 (Freezing Encoder): The shared XLM-ROBERTa encoder's weights are frozen. Only the language-specific classification heads are trained using binary cross-entropy loss. This stage focuses on adapting the heads to the specific emotion distributions of each language.
- Stage 2 (Fine-tuning): The encoder is unfrozen, and the entire model (encoder and heads) is fine-tuned with a higher learning rate. This allows the encoder to adapt more broadly while still benefiting from the specialized heads.
This approach aims to capture both universal emotional cues (via the shared encoder) and language-dependent ones (via the specialized heads), addressing the challenge of multilingual emotion detection.
Datasets
The primary dataset used is BRIGHTER, introduced in a related work (Muhammad et al., 2025a). This dataset is notable for being a comprehensive collection of multi-labeled emotion-annotated texts spanning 28 languages, with a significant focus on predominantly low-resource languages from Africa, Asia, Eastern Europe, and Latin America. This contrasts with much prior research that concentrated on high-resource languages.
Key Insights and Ideas
- Hard parameter sharing is effective for multilingual emotion detection: The architecture's success across 28 languages, including low-resource ones, highlights the power of a shared encoder for capturing commonalities in emotional expression.
- Language-specific heads are crucial for nuance: While the encoder provides a strong foundation, the specialized heads are essential for adapting to the unique ways emotions are expressed and distributed in different languages.
- Two-stage training balances generalization and specialization: Freezing the encoder initially allows for focused learning on language-specific patterns, while subsequent fine-tuning enables the model to generalize better across all languages.
- Data distribution matters significantly: The paper points out that the balance of data for each emotion class within a language directly impacts model performance, with underrepresented emotions leading to poorer detection.
- Inductive biases from the encoder play a role: Even when data distributions are similar, differences in performance between languages suggest that the inherent inductive biases learned by the encoder contribute to how well it can represent emotional cues.
Key Results
The XLM-Muriel model achieved strong performance on the BRIGHTER dataset:
- High-resource languages: Macro F1-scores up to 0.84.
- Severely imbalanced low-resource languages: Maintained robust performance with macro F1-scores around 0.63.
- Model Size Matters: Larger XLM-ROBERTa models (e.g., XLM-ROBERTa-Large) demonstrated superior cross-lingual transfer and richer inductive bias compared to smaller variants.
- Comparison with other models: The paper presents results comparing different XLM model settings (base vs. large, single vs. multi-head) and other models like LaBSE, showing that their XLM-ROBERTa-Large with multi-head setup generally performed well.
Limitations and Future Work
Limitations: * Balancing shared encoder and specialized heads: It's challenging to ensure the shared encoder fully captures unique linguistic nuances while the expert heads adequately compensate for any generic representations. This can lead to overfitting or underfitting in expert heads. * Data imbalance: The differing amounts of training data across languages can exacerbate performance inconsistencies and underrepresent certain emotional classes in low-resource scenarios. * Pre-trained model biases: XLM-ROBERTa-Large, like other large pre-trained models, may inherently carry biases towards high-resource languages, potentially hindering generalization in truly low-resource settings. * Training strategy tuning: The two-stage optimization process requires careful tuning of learning rates and might not uniformly generalize across all languages.
Future Work: * Architectural innovations: Exploring more advanced architectures to better balance shared and language-specific learning. * More balanced data collection: Addressing the issue of data imbalance by collecting more representative data for underrepresented languages and emotions. * Increasing model size: Investigating the impact of even larger models. * Fine-tuning hyperparameter optimization: More effectively tuning hyperparameters for improved performance. * Targeted fine-tuning: Applying fine-tuning strategies on specific subsets of data to further boost performance. * Detailed error analysis: Conducting in-depth comparisons with top-performing models to identify and address specific limitations.
Summary updated