Zhegu at SemEval-2023 Task 9: Exponential Penalty Mean Squared Loss for Multilingual Tweet Intimacy Analysis

Pan He, Yanru ZhangACL 2023

TL;DR

This paper introduces a novel loss function, Exponential Penalty Mean Squared Loss (EPM), designed to improve the training of regression models on challenging "hard" samples in multilingual tweet intimacy analysis. By combining EPM with language-based contrastive learning and frozen tuning, the Zhegu team achieved state-of-the-art results on the SemEval-2023 Task 9, particularly excelling in zero-shot prediction for unseen languages.

What Problem Does This Paper Address?

Understanding "text intimacy" – the degree of emotional closeness expressed in text – is crucial for analyzing social relationships and communication. However, this task presents unique challenges: intimacy is subjective and difficult to define precisely, leading to many "hard" samples that standard regression models struggle to learn. Furthermore, existing datasets and methods for multilingual intimacy analysis are scarce. This paper tackles the problem of effectively modeling tweet intimacy across multiple languages, especially for languages not seen during training, by developing a more robust regression approach.

Methods

The Zhegu team proposed three main methodological contributions:

  • Exponential Penalty Mean Squared Loss (EPM): This is the core innovation. EPM builds upon Mean Squared Error (MSE) by adding an exponential penalty term that amplifies the gradient for samples with larger prediction errors. This encourages the model to pay more attention to difficult-to-learn samples, effectively addressing the long-tail distribution of intimacy scores where extreme values are rare.
  • Language-Based Contrastive Loss (LCL): To improve multilingual understanding, they introduced a contrastive loss that pulls text embeddings from the same language closer together while pushing embeddings from different languages further apart. This helps the model leverage shared linguistic structures and distinguish language-specific nuances.
  • Frozen Tuning: To enhance generalization to unseen languages, they employed frozen tuning. This technique involves freezing a portion of the pre-trained XLM-R model's parameters (the "domain parameters") and only fine-tuning the remaining layers. This leverages the robust cross-lingual representations of the pre-trained model while allowing for adaptation to the specific task.

These methods were combined with the XLM-ROBERTa-large model as the backbone.

Datasets

The experiments utilized the SemEval-2023 Task 9 dataset, which consists of:

  • Labeled Training Set: 9,491 data points across six languages: English (en), Chinese (zh), French (fr), Italian (it), Portuguese (po), and Spanish (sp).
  • Unlabeled Test Set: 13,697 data points across ten languages, including four "new" languages (Arabic (ar), Dutch (du), Hindi (hi), and Korean (ko)) not present in the training set.

The intimacy scores in the dataset range from 1 to 5. The authors note a long-tail distribution in the labeled data, with significantly more samples at lower intimacy scores (e.g., 1.0) compared to higher scores (e.g., 5.0).

Key Insights and Ideas

  • EPM's Advantage for Hard Samples: The exponential penalty in EPM is crucial for pushing models beyond simply minimizing average error. It specifically targets outliers and difficult cases, which are prevalent in subjective tasks like intimacy analysis.
  • Language as a Signal for Contrastive Learning: Treating language as the basis for positive and negative sample pairing in contrastive learning is an effective way to build multilingual representations that respect language boundaries while fostering cross-lingual understanding.
  • Frozen Tuning for Zero-Shot Performance: Freezing parts of the pre-trained model is a practical strategy to improve generalization to entirely new languages, as it prevents catastrophic forgetting of general cross-lingual knowledge.
  • EPM's Role in Preventing Gradient Vanishing: When combined with contrastive learning, EPM proved more robust than standard MSE, helping to avoid gradient vanishing issues that can occur when optimizing multiple, potentially conflicting, loss functions.

Key Results

The Zhegu system achieved strong performance across various evaluation settings:

  • Overall Performance: The best performing model, an ensemble of EPM, CL, and FT, achieved a Pearson correlation coefficient (PCC) of 72.00 on the "Seen Languages" test set and 46.92 on the "Unseen Languages" test set.
  • EPM vs. MSE: EPM consistently outperformed MSE on both validation and test sets, showing improvements of around 2% on seen languages and 7% on unseen languages. Figure 4 also demonstrates that EPM converges faster and with less fluctuation than MSE.
  • Contrastive Learning Boost: Adding contrastive learning (EPM + CL) further improved performance, particularly on unseen languages, achieving a 2% promotion over EPM alone.
  • Frozen Tuning Contribution: Frozen tuning also provided a notable improvement of 1% on unseen languages, highlighting its effectiveness for generalization.
  • State-of-the-Art: The ensemble method achieved the highest reported scores for both seen and unseen languages in the competition.

Limitations and Future Work

  • Optimization Complexity: The authors note that simultaneously optimizing EPM and contrastive loss can lead to slower initial convergence and potential gradient vanishing issues, though EPM mitigates this.
  • Ensemble Necessity: While the ensemble achieved the best results, it implies that individual components might not reach peak performance in isolation.
  • Future Work: The authors suggest exploring the effectiveness of EPM in other natural language regression tasks, indicating its potential as a general-purpose loss function for challenging regression problems.

Summary updated

BibTeX
@inproceedings{he2023zhegu,
  author = {Pan He and Yanru Zhang},
  title = {{Zhegu at SemEval-2023 Task 9: Exponential Penalty Mean Squared Loss for Multilingual Tweet Intimacy Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure