UAlberta at SemEval-2024 Task 1: A Potpourri of Methods for Quantifying Multilingual Semantic Textual Relatedness and Similarity

Ning Shi, Senyu Li, Guoqing Luo, Amirreza Mirzaei, Ali Rafiei, Jai Riley, Hadi Sheikhi, Mahvash Siavashpour, Mohammad Tavakoli, Bradley Hauer, Grzegorz KondrakNAACL 2024

TL;DR

This paper presents a comprehensive study on Semantic Textual Relatedness (STR) across 14 languages, exploring the relationship between similarity and relatedness and whether relatedness is preserved across translations. The authors developed a diverse set of methods, including explicit semantic analysis, extrinsic task reduction, distributional embeddings, and large language models (LLMs), finding that ensembling these approaches yields state-of-the-art results, particularly for multilingual and cross-lingual scenarios.

What Problem Does This Paper Address?

The core challenge this paper tackles is accurately quantifying how semantically related two sentences are, especially across different languages. While semantic similarity (e.g., "the cat sat on the mat" and "a feline rested on the rug") is a well-studied area, semantic relatedness is broader, encompassing relationships like antonymy (e.g., "it is raining" vs. "it is not raining"), which have low similarity but high relatedness. Traditional methods often conflate these two. Furthermore, understanding how these relationships translate across languages is crucial for building truly multilingual NLP systems. The authors investigate two key hypotheses: (1) that semantic similarity is a specific instance of semantic relatedness, and (2) that semantic relatedness remains consistent even when sentences are translated into different languages.

Methods

The UAlberta team explored a "potpourri" of methods, categorized as follows:

  • Explicit Semantic Methods: These methods rely on linguistic resources.
  • Concept Overlap: Uses WordNet senses to identify shared lexical concepts between sentences.
  • AMR Similarity: Leverages Abstract Meaning Representations (AMRs) to capture sentence meaning as graphs and then computes similarity using the Smatch F1-score.
  • Extrinsic Methods: These reduce the STR task to other established NLP tasks.
  • Paraphrase Identification (PI): Treats STR as a binary classification problem to determine if one sentence is a paraphrase of another, using a fine-tuned RoBERTa model.
  • Textual Entailment (NLI): Uses Natural Language Inference models to assess if one sentence can be inferred from another, averaging probabilities from both directions.
  • Distributional Methods: These utilize dense vector representations of sentences.
  • Embeddings: Employs pre-trained LLMs (BERT and RoBERTa) to generate sentence embeddings, then calculates cosine similarity between them.
  • Large Language Models (LLMs):
    • Prompting: Uses few-shot in-context learning with ChatGPT to estimate STR.
  • Fusion: Combines contextualized embeddings from multiple LLMs and trains a linear layer to predict STR.
  • Fine-tuning: Adds a linear regression head to pre-trained LLMs (T5, GPT-2, RoBERTa, MPNet) and fine-tunes them on STR data.
  • Ensemble Modules: The core innovation lies in combining individual methods.
  • Linear Combination: A simple average of normalized scores from multiple methods.
  • Supervised Regression: Uses an XGBoost regressor to learn how to best combine the scores from individual methods, treating each score as a feature.

Datasets

The primary dataset used is SemRel2024 (Ousidhoum et al., 2024a), which features sentence pairs annotated with relatedness scores across 14 languages. To test their hypotheses, the authors also created new cross-lingual datasets by translating parts of the SemRel2024 development set into Spanish. Additionally, they evaluated their methods on SemEval 2017 Task 1 STS datasets (Cer et al., 2017) to compare performance on similarity vs. relatedness tasks.

Key Insights and Ideas

  • Similarity as a Special Case of Relatedness: The experiments provide empirical support for the hypothesis that highly similar sentences are also highly related. This is evidenced by the strong correlation observed between STR and STS task results.
  • Relatedness is Preserved Across Translation: The cross-lingual experiments suggest that semantic relatedness scores remain largely consistent when sentences are translated, supporting the idea that the underlying semantic relationships are robust to language shifts.
  • Ensembling is Key: Combining multiple diverse methods, especially through supervised regression, significantly boosts performance over individual models, highlighting the benefit of leveraging different strengths.
  • LLMs Show Promise but Need Fine-tuning: While prompting LLMs like ChatGPT offers a baseline, fine-tuning them on specific STR tasks, or using fusion techniques, yields much better results.
  • Explicit Semantic Methods Vary in Efficacy: Concept overlap performs comparably to word overlap, but AMR-based similarity lags, suggesting that graph representations might not always capture relatedness effectively or that their quality is insufficient.

Key Results

  • The UAlberta team achieved highly competitive results, ranking among the top three entries in 16 out of 51 language/track settings.
  • Their best-performing supervised ensemble system, Target-XGB, achieved the highest score in the English Track A among teams submitting a system description paper.
  • The Target-XGB system, which fine-tunes individual models on translated target language data and incorporates English data, consistently outperformed other ensemble and individual methods across most languages.
  • For cross-lingual tasks, methods like Prompt and Fusion showed strong performance when applied directly to multilingual data, while others like WordOvlap struggled without translation.

Limitations and Future Work

  • Translation Noise: The authors acknowledge that machine translation can introduce noise, which might affect cross-lingual performance. Their data augmentation techniques aim to mitigate this.
  • LLM Limitations: While LLMs show promise, the paper notes that ChatGPT's STR capabilities are still limited, and there's considerable scope for improvement.
  • AMR Representation Quality: The underperformance of AMR similarity suggests that the quality or suitability of AMR representations for relatedness tasks needs further investigation.
  • Future Work: The paper implicitly suggests further exploration into more robust cross-lingual transfer learning techniques, improved LLM fine-tuning strategies, and deeper analysis of why certain explicit semantic methods perform better than others. The creation of new cross-lingual datasets also paves the way for future research in this area.

Summary updated

BibTeX
@inproceedings{shi2024ualberta,
  author = {Ning Shi and Senyu Li and Guoqing Luo and Amirreza Mirzaei and Ali Rafiei and Jai Riley and Hadi Sheikhi and Mahvash Siavashpour and Mohammad Tavakoli and Bradley Hauer and Grzegorz Kondrak},
  title = {{UAlberta at SemEval-2024 Task 1: A Potpourri of Methods for Quantifying Multilingual Semantic Textual Relatedness and Similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure