TL;DR
This paper investigates the effectiveness of Sentence-BERT (SBERT) for Semantic Textual Relatedness (STR) across various low-resource languages. The authors found that while SBERT shows promise, the choice of pre-trained language models (PLMs) significantly impacts performance, with monolingual PLMs not always outperforming multilingual ones, especially in resource-scarce scenarios.
What Problem Does This Paper Address?
Semantic Textual Relatedness (STR) is a crucial NLP task that measures how closely two pieces of text relate in meaning. Unlike semantic similarity (which focuses on synonyms or entailment), STR encompasses a broader range of lexical and semantic relationships. However, most research and available datasets have historically focused on high-resource languages like English, leaving a significant gap in understanding and improving STR for low-resource languages. This paper addresses this gap by evaluating SBERT's performance on STR for several low-resource languages, exploring the impact of different pre-trained language models (PLMs) on this task.
Methods
The core of the methodology revolves around fine-tuning the Sentence-BERT (SBERT) architecture. SBERT, unlike standard BERT, uses a Siamese network structure where two identical BERT models share weights to process sentence pairs. This architecture is well-suited for tasks involving sentence pairs, such as STR. The authors employed a regression task, where the cosine similarity between the sentence embeddings produced by SBERT is used to predict the relatedness score.
They experimented with various pre-trained language models (PLMs) to fine-tune SBERT. These included:
- Multilingual PLMs: LaBSE (Language-agnostic BERT Sentence Embedding) and
paraphrase-multilingual-mpnet-base-v2(pp-mpnet-v2). These models are trained on a large number of languages and are designed for cross-lingual tasks. - Language-Specific Monolingual PLMs: For certain languages, they utilized PLMs specifically trained on that language's data. Examples include MahaSBERT and TeluguSBERT for Marathi and Telugu, and DziriBERT for Algerian Arabic.
- Specialized Multilingual PLMs: AfroXLMR was used for African languages, and IndicSBERT for Indian languages.
The models were trained on the SemRel2024 dataset and evaluated using the Spearman correlation coefficient, a standard metric for measuring the agreement between predicted and human-annotated relatedness scores.
Datasets
The primary dataset used in this study is the SemRel2024 dataset, specifically Track A, which focuses on semantic textual relatedness for African and Asian languages. This dataset contains sentence pairs annotated with fine-grained relatedness scores for 9 languages, 7 of which are considered low-resource. The authors used the training and development splits for fine-tuning and evaluation during development, and the combined training+development data for final testing.
Key Insights and Ideas
- SBERT is a viable architecture for low-resource STR: The study demonstrates that SBERT can be effectively fine-tuned for STR tasks even in low-resource settings, achieving competitive results.
- PLM choice is critical: The performance of SBERT is highly dependent on the underlying pre-trained language model. This highlights the importance of selecting appropriate PLMs for specific languages and tasks.
- Monolingual vs. Multilingual is not a clear winner: Contrary to expectations, monolingual PLMs did not consistently outperform multilingual PLMs for Marathi and Telugu. This suggests that the availability and quality of monolingual data and pre-training are crucial factors, and sometimes a well-trained multilingual model can be more effective.
- Resource scarcity is a major hurdle: For languages like Moroccan Arabic and Hausa, even specialized multilingual models like AfroXLMR did not perform as well as expected. This points to a broader issue of insufficient pre-trained language resources for many low-resource languages.
- Task-specific pre-training matters: The performance of AfroXLMR for African languages and DziriBERT for Algerian Arabic suggests that models pre-trained with objectives closer to the target task (like sentence embeddings or dialect-specific language modeling) can yield better results.
Key Results
The paper reports the following key results:
- The models for English, Algerian Arabic, and Kinyarwanda ranked 12th, 5th, and 8th respectively in their respective categories.
- Overall, the team's submissions ranked 5th out of 40 submissions in Track A, achieving an average Spearman correlation of 0.74.
- For English, the
sentence-t5-largemodel showed strong performance. - For Spanish, LaBSE performed well, scoring close to the baseline.
- In Marathi and Telugu, monolingual models (MahaSBERT, TeluguSBERT) performed well during development but did not surpass the baseline during testing.
- For Moroccan Arabic and Hausa, multilingual models scored lower than the baseline during testing, indicating a lack of robust pre-trained models for these languages.
Limitations and Future Work
- Limited availability of monolingual PLMs: The study highlights a significant drawback: the scarcity of good, publicly available monolingual pre-trained language models for many low-resource languages. This limits the ability to leverage monolingual data effectively.
- Performance variability: The performance of models varied significantly across languages, underscoring the challenge of low-resource NLP.
- Future Work: The authors suggest that the availability of better pre-trained language models, especially for low-resource languages, is crucial. Further research could focus on developing more effective cross-lingual transfer learning techniques and creating more comprehensive datasets for semantic textual relatedness in these languages. The paper also implies a need for more investigation into why monolingual models don't always outperform multilingual ones in these specific contexts.
Summary updated