BITS Pilani at SemEval-2024 Task 1: Using text-embedding-3-large and LaBSE embeddings for Semantic Textual Relatedness

Dilip Venkatesh, Sundaresan RamanNAACL 2024

TL;DR

This paper tackles the challenge of measuring how semantically related two pieces of text are, a crucial task for many NLP applications. The authors propose a zero-shot approach using powerful pre-trained embedding models, specifically OpenAI's text-embedding-3-large and Google's LaBSE, achieving competitive results on the SemEval-2024 Task 1.

What Problem Does This Paper Address?

Understanding the meaning of text goes beyond just recognizing individual words; it involves grasping how words and sentences connect and relate to each other. Semantic relatedness quantifies this connection, measuring how closely two pieces of text (like sentences or words) align in meaning. This is fundamental for tasks such as question answering, text summarization, and information retrieval, where understanding the nuances of meaning is paramount. While semantic similarity focuses on specific lexical relationships (synonyms, antonyms), semantic relatedness is broader, encompassing any form of meaningful connection. The SemEval-2024 Task 1 specifically aimed to evaluate systems on their ability to predict this relatedness score across various languages, including low-resource ones.

Methods

The core of this paper's approach is a zero-shot learning strategy. This means they leverage pre-trained language models without any task-specific fine-tuning on the training data. They use two primary embedding models:

  • OpenAI's text-embedding-3-large: This is a state-of-the-art, large-scale embedding model known for its high-dimensional embeddings (3072 dimensions) and strong performance on various benchmarks.
  • LaBSE (Language-agnostic BERT Sentence Embedding): Developed by Google, LaBSE is specifically designed to generate language-agnostic sentence embeddings, making it particularly useful for multilingual tasks, especially for languages with fewer resources.

For each sentence pair, they generate embeddings using one of these models. The relatedness score is then calculated by comparing these embeddings. While the paper mentions dot product as a potential metric, it notes that normalized dot product or cosine similarity is often preferred to avoid bias from vector length. The final prediction is a single score representing the semantic relatedness.

Datasets

The primary dataset used for evaluation is the SemRel datasets for SemEval-2024 Task 1. This dataset is notable for its annotation across 14 languages, including English and several low-resource languages like Algerian Arabic, Amharic, Hausa, Kinyarwanda, Marathi, Moroccan Arabic, Spanish, and Telugu. The dataset consists of sentence pairs with manually annotated relatedness scores ranging from 0 (completely unrelated) to 1 (maximum relations). The authors also mention using the training portion of this dataset for evaluating their models' performance on different languages, as seen in Table 2.

Key Insights and Ideas

  • The Power of Pre-trained Embeddings: The paper strongly demonstrates that powerful, general-purpose pre-trained embedding models can achieve impressive results on semantic relatedness tasks even without task-specific fine-tuning. This highlights the transferability of knowledge learned by these large models.
  • LaBSE's Multilingual Advantage: LaBSE proves to be a valuable tool for low-resource languages, providing good performance where other models might struggle due to data scarcity.
  • Zero-Shot is Viable: The success of a zero-shot approach suggests that for many semantic tasks, the inherent understanding embedded within large language models is sufficient, reducing the need for extensive labeled data for every new task or language.
  • Model Selection Matters: The authors strategically chose text-embedding-3-large for some languages and LaBSE for others based on their observed performance on the training data, indicating that the best model can be language-dependent.

Key Results

The paper reports Spearman rank correlation coefficients as the evaluation metric, comparing their system's predictions against human judgments.

  • For English, their system using text-embedding-3-large achieved a score of 0.832, surpassing the baseline score.
  • Across other languages, their zero-shot approach yielded competitive results. For instance, they achieved 0.800 for Amharic and 0.814 for Telugu using LaBSE, and 0.509 for Algerian Arabic using text-embedding-3-large.
  • Table 2 shows a performance comparison on the training set, where text-embedding-3-large generally outperformed LaBSE on English and some other languages, while LaBSE showed strength in others.

Limitations and Future Work

  • Open-Source vs. Proprietary Models: A significant limitation acknowledged is that text-embedding-3-large is not an open-source model. This raises concerns about reproducibility and potential inherent biases within proprietary models.
  • Inherent Biases: The authors explicitly mention that these models may contain inherent biases, which could affect their performance and fairness across different linguistic and cultural contexts.
  • Zero-Shot vs. Fine-Tuning: While the zero-shot approach is effective, it's possible that fine-tuning these models on task-specific data could yield even better results, especially for languages with more available training data. Future work could explore this avenue.
  • Model Selection Strategy: The paper's strategy for selecting between text-embedding-3-large and LaBSE was based on training set performance. Further investigation into more robust or automated model selection methods could be beneficial.

Summary updated

BibTeX
@inproceedings{venkatesh2024bits,
  author = {Dilip Venkatesh and Sundaresan Raman},
  title = {{BITS Pilani at SemEval-2024 Task 1: Using text-embedding-3-large and LaBSE embeddings for Semantic Textual Relatedness}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure