TL;DR
This paper tackles the challenge of evaluating Semantic Textual Relatedness (STR) across multiple languages, moving beyond English-centric NLP. By leveraging the SemEval-2024 shared task and employing various transformer-based models, the authors demonstrate promising results in supervised, unsupervised, and cross-lingual settings, paving the way for more robust multilingual NLP applications.
What Problem Does This Paper Address?
The explosion of online content necessitates robust Natural Language Processing (NLP) that can understand nuances across diverse languages. Semantic Textual Relatedness (STR) is crucial for this, going beyond simple word overlap to capture deeper connections between sentences, including topic, sentiment, and perspective. However, most prior NLP research has been heavily focused on English, leaving a significant gap in understanding and evaluating STR for other languages, especially low-resourced ones. This limitation hinders applications like multilingual information retrieval and machine translation. This paper aims to bridge this gap by exploring multilingual STR, specifically for English, Marathi, Hindi, and Spanish.
Methods
The authors participated in the SemEval-2024 Task 1, which offered three tracks to evaluate STR techniques:
- Supervised Learning (Track A): This track involved training models on labeled datasets. The approach utilized both traditional machine learning models like Support Vector Regression (SVR) and XGBoost, alongside advanced sentence-transformer-based models such as
all-mpnet-base-v2andmarathi-sentence-bert-nli. The latter were chosen to compensate for smaller corpora by leveraging pre-trained models on larger datasets. Feature extraction involved Term Frequency-Inverse Document Frequency (TF-IDF). - Unsupervised Learning (Track B): Here, the challenge was to develop systems that learn semantic relationships without labeled data. The models used were BERT-based uncased and
hindi-bert-v2. The core algorithm involved initializing the BERT model, calculating sentence embeddings for both sentences in a pair, and then computing the cosine similarity between these embeddings. - Cross-lingual Learning (Track C): This track required leveraging knowledge from a source language (English) to address a target language (Hindi/Spanish) with limited resources. The process involved translating sentences from one language to another using a deep translation API. Then, sentence-transformer models (
all-mpnet-base-v2andhindi-sentence-bert-nli) were initialized and trained on the translated datasets. The encoded sentence pairs were then used to calculate similarity scores.
Datasets
The primary dataset used was the SemRel2024 Dataset (Ousidhoum et al., 2024a). This dataset is notable for its multilingual nature, covering 14 diverse languages. For this work, the authors focused on English, Hindi, Marathi, and Spanish. Each entry in the dataset consists of a sentence pair and a semantic similarity score ranging from 0 to 1. The dataset was split into training, development, and testing sets for each track and language.
Key Insights and Ideas
- Language-specific models show promise: The results suggest that language-specific models (e.g.,
marathi-sentence-bert-nli) can outperform even multilingual models on their respective languages, highlighting the importance of specialized training data. - Transformer models dominate: Across supervised and unsupervised tracks, sentence-transformer-based models consistently outperformed traditional methods like SVR and XGBoost, underscoring their effectiveness in capturing nuanced semantic relationships.
- Cross-lingual translation is viable: The cross-lingual track demonstrated that by translating datasets, it's possible to leverage existing models and achieve reasonable performance in languages with less available labeled data.
- SemEval's role in advancing multilingual NLP: The SemEval-2024 shared task provided a crucial platform for evaluating and pushing the boundaries of multilingual STR, encouraging research in under-resourced languages.
Key Results
- Track A (Supervised): BERT-based models achieved significantly higher correlation scores than SVR and XGBoost in both English and Marathi. The
marathi-nlimodel achieved a correlation score of 0.871 for Marathi. - Track B (Unsupervised): In English,
BERT-base-uncasedoutperformedsentence-t5. In Hindi,hindi-bert-v2achieved a higher correlation coefficient (0.796) compared toIndic-BERT, despite similar F1 scores. - Track C (Cross-lingual): The models achieved worthy correlation scores, with
all-mpnet-base-v2for Spanish-to-English scoring 0.82 andhindi-sentence-bert-nlifor English-to-Hindi scoring 0.71. - The paper achieved 1st place in Unsupervised Learning for Hindi.
Limitations and Future Work
- Dataset Size: While the SemRel2024 dataset is comprehensive, the paper acknowledges that larger datasets and broader language coverage are crucial for further advancements in multilingual NLP.
- External Datasets: The authors note that they did not utilize any external datasets, which might have influenced performance compared to systems that do.
- Future Work: The authors aim to inspire further research in multilingual STR, particularly for low-resourced languages. They believe that expanding language coverage and dataset size holds immense potential for unlocking deeper understanding and empowering cross-cultural communication.
Summary updated