TL;DR
This paper tackles the challenge of accurately measuring semantic textual relatedness (STR) across multiple languages, especially low-resource ones. The authors demonstrate that fine-tuning transformer-based models significantly outperforms traditional ML and DL approaches, achieving state-of-the-art results on the SemEval-2024 Task 1 dataset for English, Marathi, Telugu, and Spanish.
What Problem Does This Paper Address?
Semantic Textual Relatedness (STR) is a fundamental task in Natural Language Processing (NLP) that aims to quantify how similar two sentences are in meaning. This capability is crucial for a wide range of applications, from search engines and chatbots to plagiarism detection and text generation. While significant progress has been made in high-resource languages like English, achieving comparable performance in low-resource languages (LRLs) like Marathi, Telugu, and Spanish remains a considerable challenge. Existing methods often struggle with the nuances of these languages, sometimes resorting to suboptimal translation-based approaches or failing to capture the full meaning of phrases and idioms. This paper addresses this gap by exploring and evaluating various machine learning, deep learning, and transformer-based models for STR across four languages, with a particular focus on improving performance in LRLs.
Methods
The study employed a multi-pronged approach, evaluating three main categories of models:
- Machine Learning (ML) Approaches: Traditional ML models like Support Vector Regression (SVR) and Random Forest (RF) were used. These models typically rely on feature engineering, such as TF-IDF, to represent text.
- Deep Learning (DL) Approaches: A Siamese Neural Network (SNN) architecture was implemented for the English dataset. This approach uses word embeddings (Word2Vec, FastText) and LSTM layers to learn sentence representations and compute similarity using cosine embedding loss.
- Transformer-based Models: This was the core focus of the study. The authors fine-tuned several pre-trained transformer models, including:
- English: all-MiniLM-L6-v2, all-MiniLM-L12-v2, Mpnet-v2, Distilbert, and others.
- Marathi: Marathi-SBERT, Marathi-BERT-nli, and MarathiRoberta.
- Telugu: Telugu-SBERT, Telugu-BERT-nli, and LaBSE.
- Spanish: Spanish-BERT, Roberta-bne, and stsb-xlm-r-multilingual.
The transformer models were fine-tuned on the specific task datasets for each language. The study also explored various hyperparameter settings for these models, including learning rate, weight decay, and warmup steps.
Datasets
The primary dataset used for this study was developed by Ousidhoum et al. (2024a) for SemEval-2024 Task 1. This dataset includes labeled and unlabeled data for four languages: English, Marathi, Telugu, and Spanish. The labeled data consists of sentence pairs with a score ranging from 0 to 1, indicating their semantic relatedness. The dataset is divided into development (dev), test, and training sets for each language.
| Language | Dev set | Test set | Train set |
|---|---|---|---|
| English | 250 | 2600 | 11000 |
| Marathi | 294 | 299 | 2400 |
| Telugu | 130 | 297 | 2340 |
| Spanish | 140 | 600 | 1562 |
| Total | 814 | 3796 | 17302 |
Key Insights and Ideas
- Transformer Dominance: Transformer-based models, particularly when fine-tuned, overwhelmingly outperform traditional ML and DL models for semantic textual relatedness across all tested languages. This highlights their superior ability to capture contextual nuances.
- Low-Resource Language Performance: The study successfully demonstrates that fine-tuned transformers can achieve strong performance even in low-resource languages, bridging a significant gap in NLP research.
- Bidirectional Context is Key: The paper suggests that the success of transformers lies in their ability to capture bidirectional context within sentences, leading to more nuanced and accurate representations of meaning.
- Pre-training Matters: The pre-trained nature of these transformer models, trained on massive datasets for various linguistic tasks, provides a strong foundation that significantly boosts performance after fine-tuning.
Key Results
The transformer-based models achieved impressive results, significantly outperforming ML and DL approaches. The paper reports the following top Spearman correlation coefficients (ρ) for each language:
- English: MiniLM-L6 achieved a ρ of 0.822.
- Marathi: MarathiSBERT achieved a ρ of 0.870.
- Telugu: Telugu-BERT-nli achieved a ρ of 0.820.
- Spanish: Spanish-BERT achieved a ρ of 0.677.
These results led to the team's rankings in the SemEval-2024 Task 1: 22nd for English, 11th for Marathi, 11th for Telugu, and 14th for Spanish.
Limitations and Future Work
- Limitations:
- The paper notes that ML models performed poorly, and DL models showed only slight improvements, indicating room for further development in these areas.
- The study acknowledges that while transformer models excel, there are still challenges in accurately predicting scores for sentences with significant length disparities or subtle semantic differences.
- Future Work:
- The authors suggest that augmenting training data could further enhance the performance of current models.
- Leveraging advanced techniques like Large Language Models (LLMs) and Generative Pre-trained Transformers (GPT) is proposed as a promising direction for future improvements in semantic textual relatedness tasks.
Summary updated