TL;DR
This paper presents a straightforward yet effective approach to evaluating semantic textual relatedness across 14 languages for SemEval-2024 Task 1. By fine-tuning a BERT model on English data and then applying it to machine-translated versions of other languages, the authors achieved competitive results, particularly for low-resource languages, demonstrating the power of leveraging existing high-resource language models.
What Problem Does This Paper Address?
The paper tackles the challenge of evaluating semantic textual relatedness (STR) across a diverse set of languages, with a particular focus on low-resource languages. While significant progress has been made in STR for English, developing robust models for languages with limited data and NLP resources is a major hurdle. This task aims to bridge this gap by providing a benchmark and encouraging research into methods that can scale effectively to languages with fewer available tools and datasets. The authors highlight that progress in STR for low-resource languages is crucial for advancements in areas like information retrieval, summarization, and the development of Large Language Models (LLMs).
Methods
The core of the OZemi system is a hybrid approach that combines traditional NLP techniques with advanced deep learning models. Initially, they explored classic methods like TF-IDF and cosine similarity with different word embeddings (binary occurrence vectors and TF-IDF transformer-based vectors) on the English baseline. For the main STR evaluation, they leveraged fine-tuned BERT models.
The key innovation lies in their strategy for handling non-English languages:
- English Baseline: They fine-tuned BERT models on the English dataset, achieving a strong performance.
- Machine Translation: For the other 13 languages, they translated the training datasets into English using the Google Translate API. This allowed them to apply their well-performing English BERT model to all languages, maintaining consistency and avoiding the need to train language-specific models from scratch.
- Multilingual Models: They also experimented with multilingual BERT models, but found that the machine-translation approach generally yielded better results, except for Spanish where a multilingual model performed better.
Datasets
The primary dataset used is the SemEval-2024 Task 1 dataset, adapted from the STR-2022 dataset. This dataset contains sentence pairs manually annotated with relatedness scores from 0 to 1. The task includes data for 14 languages, with Track A focusing on supervised subtasks using provided labeled datasets. The paper specifically mentions data for Algerian Arabic, Amharic, English, Hausa, Kinyarwanda, Marathi, Moroccan Arabic, Spanish, and Telugu for Track A. The non-English datasets for Track A are noted to be significantly smaller than the English dataset.
Key Insights and Ideas
- Leveraging High-Resource Models: The paper strongly suggests that fine-tuning models on high-resource languages (like English) and then applying them to machine-translated data from low-resource languages can be a surprisingly effective and cost-efficient strategy.
- Machine Translation as a Data Augmentation Tool: Machine translation, particularly with tools like Google Translate, can serve as a powerful method for data augmentation, enabling the application of robust models to languages with limited resources.
- The Nuance of "Relatedness" vs. "Similarity": The authors acknowledge the distinction between semantic similarity (lexical overlap) and semantic relatedness (broader conceptual connections), noting that while the task is relatedness, some datasets used in its construction lean towards similarity.
- Google Translate's Effectiveness: Google Translate was found to be a suitable choice for this task, potentially due to its tendency to favor common words, which can simplify sentences and artificially increase relatedness scores, making them easier for models to capture.
- Challenges with Arabic Dialects: The approach struggled with Algerian and Moroccan Arabic, likely due to Google Translate's limitations in handling these specific dialects, forcing reliance on general Arabic and potentially obfuscating semantic links.
Key Results
The authors' simplistic approach achieved middle-of-the-pack ranks in most languages. They secured top-third rankings for English, Marathi, and Spanish. The English baseline model, using a fine-tuned BERT, achieved an official score of 0.81, ranking 10th out of 34 submissions. The MT+English model generally outperformed language-specific models for non-English languages, with scores for most languages being less than 0.5, except for Spanish (0.67).
Limitations and Future Work
- Limitations:
- The approach struggled with Arabic dialects (Algerian and Moroccan Arabic) due to the limitations of Google Translate's general Arabic translation.
- The authors, lacking native speakers for most languages, found manual evaluation of machine-translated output difficult.
- For Spanish, the machine-translation approach yielded lower scores than using a multilingual model, suggesting translation might sometimes obscure semantic links.
- Future Work:
- The authors suggest exploring mixed methods, starting with language-specific models and then incorporating machine translation and larger English-trained models.
- Investigating why machine translation might negatively impact relatedness scores for languages more closely related to English (like Spanish) is a promising direction.
Summary updated