UMBCLU at SemEval-2024 Task 1: Semantic Textual Relatedness with and without machine translation

Shubhashis Roy Dipta, Sai VallurupalliNAACL 2024

TL;DR

This paper tackles the challenge of measuring semantic textual relatedness (STR) across a diverse set of African and Asian languages, many of which are low-resource. The authors propose two novel models, TranSem and FineSem, leveraging large language models (LLMs) and exploring the impact of machine translation and direct fine-tuning, achieving competitive results and even outperforming baselines on several languages.

What Problem Does This Paper Address?

Understanding the semantic relationship between sentences is a fundamental task in Natural Language Processing (NLP), with applications ranging from question answering to text summarization. However, most research and datasets have historically focused on high-resource languages like English. The SemEval-2024 Task 1 specifically aims to bridge this gap by focusing on Semantic Textual Relatedness (STR) for 14 African and Asian languages, many of which have limited linguistic resources. This presents a significant challenge: how to build effective STR models for languages with scarce training data, and how to best leverage existing powerful models (LLMs) in this low-resource setting. The task is further complicated by the need to evaluate models under different training paradigms: supervised, unsupervised, and cross-lingual.

Methods

The authors developed two main models:

  • TranSem: This model utilizes a Siamese network architecture, inspired by previous work on sentence embeddings. It encodes sentence pairs using a shared sentence encoder and then uses a cosine similarity loss to predict the relatedness score. TranSem leverages sentence embeddings from models like DistilRoberta and Sentence-T5. A key aspect of TranSem is its exploration of machine translation: sentences from low-resource languages are translated into English, and then the model operates on these translated pairs. This approach allows for data augmentation and leverages the strengths of English-centric LLMs.
  • FineSem: This model takes a direct fine-tuning approach using the T5 transformer architecture. T5, pre-trained on a variety of tasks including semantic textual similarity (STS), is fine-tuned on the STR task. The authors experiment with different configurations: fine-tuning T5 models individually for each language, a single unified model for all languages, and a single model trained on translated and augmented data. This model directly contrasts the effectiveness of sentence embeddings (TranSem) with end-to-end fine-tuning.

Both models explore different training strategies and the impact of machine translation. For cross-lingual evaluation, a T5 model fine-tuned on English STR data is used.

Datasets

The primary dataset used is from SemEval-2024 Task 1, which provides sentence pairs annotated for relatedness across 14 African and Asian languages. These languages are grouped into 5 language families. The authors also leverage existing English datasets, particularly for fine-tuning the T5 models and for translating the low-resource language data. They specifically mention using Meta's "No Language Left Behind (NLLB)" models for translation.

Key Insights and Ideas

  • Machine Translation as a Data Augmentation Tool: Translating sentences from low-resource languages into English proved to be a valuable strategy. For some languages, this approach led to better performance than working directly with the original, limited data.
  • Direct Fine-tuning vs. Sentence Embeddings: The paper shows that direct fine-tuning with T5 (FineSem) can be competitive with, and sometimes outperform, approaches relying solely on sentence embeddings (TranSem), especially when using translated and augmented data.
  • Unified Models Can Be Effective: The authors demonstrate that a single T5 model fine-tuned on translated data can achieve good performance across multiple languages, highlighting the potential of unified models for low-resource scenarios.
  • Language-Specific Nuances: While translation helps, the performance varies across languages, suggesting that the effectiveness of translation and direct fine-tuning can be language-dependent.

Key Results

  • Supervised Setting (Track A): The FineSem model outperformed the official baseline for 3 languages and performed on par for another 4.
  • Cross-lingual Setting (Track C): The TranSem model achieved better performance than the baseline for 3 languages, securing 1st place for Afrikaans and 2nd place for Indonesian. It performed on par for 2 languages but struggled with the remaining 7.
  • Effectiveness of Translation: For some languages, translating to English and then applying STR models led to improved results.
  • Batch Size and Pooling: The paper provides analysis on the optimal batch size and pooling mechanisms (mean pooling being favored) for the TranSem model.

Limitations and Future Work

  • Reliance on Pre-trained LLMs: The authors acknowledge that their work relies on pre-trained LLMs, which can inherit and propagate biases present in their training data. They did not explicitly control for these biases.
  • Dataset Biases: The STR datasets themselves may contain biases related to social groups, cultural groups, race, gender, and perceptual differences of annotators, which were not controlled for.
  • Exclusion of Recent Models: Due to computational constraints, the authors did not experiment with more recent, potentially higher-performing models like Mistral-7B.
  • Future Work: The authors intend to explore prompting for STR and investigate prompt-based LLMs for translation.

Summary updated

BibTeX
@inproceedings{dipta2024umbclu,
  author = {Shubhashis Roy Dipta and Sai Vallurupalli},
  title = {{UMBCLU at SemEval-2024 Task 1: Semantic Textual Relatedness with and without machine translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure