TL;DR
This paper tackles the challenging task of Semantic Textual Relatedness (STR) for Arabic dialects, a less explored area due to data scarcity. The authors propose a novel approach combining generative data augmentation with BERT-based models for both supervised and unsupervised STR tasks, achieving top ranks in the SemEval-2024 challenge.
What Problem Does This Paper Address?
Semantic Textual Relatedness (STR) is a nuanced task that goes beyond simple semantic similarity. It assesses how two pieces of text are related across various dimensions, including topical similarity, conceptual overlap, and contextual coherence. While semantic similarity has been extensively studied, STR is less explored, particularly for low-resource languages and dialects, due to a lack of comprehensive datasets. This paper addresses this gap by participating in the SemRel-2024 shared task, which aims to advance STR research across fourteen languages, including Arabic dialects like Algerian, Moroccan, and Modern Standard Arabic (MSA). The challenge lies in developing models that can effectively capture these complex relationships, especially when labeled data is limited or unavailable.
Methods
The authors employed two main strategies for the SemRel-2024 task, tailored to the supervised (Track A) and unsupervised (Track B) settings:
- Supervised Track (A): For Algerian and Moroccan dialects, where labeled data was available, they fine-tuned BERT-based models (AraBERTv2 and ArBERTv2) using a Mean Squared Error (MSE) regression objective. A key innovation here is the generative data augmentation using Google Gemini. They prompted Gemini with specific templates to generate new sentence pairs that mimicked the style and meaning of existing data, effectively expanding the training set. This augmentation proved crucial for improving performance, especially for the Moroccan dialect.
- Unsupervised Track (B): For MSA, where labeled training data was prohibited, they relied on encoder-based scoring. Specifically, they used BERT-based models (AraBERTv2) to generate sentence embeddings and then calculated the cosine similarity between these embeddings to estimate the relatedness score. Average pooling was used on the final embedding layer to obtain sentence representations.
Both approaches involved pre-processing text using arabert_prep and concatenating sentence pairs with special tokens ([CLS] Sentence1 [SEP] Sentence2 [SEP]) before feeding them into the models.
Datasets
The paper utilized datasets from the SemRel-2024 shared task, focusing on three Arabic variants:
- Algerian Arabic: Provided with training, development, and testing splits.
- Moroccan Arabic: Also provided with training, development, and testing splits. The authors augmented the training data for this dialect.
- Modern Standard Arabic (MSA): Provided with testing and development splits, but no labeled training data, making it suitable for the unsupervised track.
The augmentation process for Moroccan Arabic resulted in an additional 757 sentence pairs, bringing the total training data to 1,681 instances.
Key Insights and Ideas
- Generative Augmentation is Powerful: The use of Google Gemini for data augmentation significantly boosted performance in the supervised track, particularly for the Moroccan dialect. This highlights the potential of large language models to create valuable training data for low-resource NLP tasks.
- Dialectal Nuances Matter: The performance differences between models and dialects suggest that specific linguistic characteristics of each dialect require tailored approaches or more extensive data. The authors noted that their models performed better on Moroccan than Algerian, potentially due to the augmentation strategy.
- Unsupervised STR is Challenging but Feasible: Achieving competitive results in the unsupervised MSA track demonstrates that encoder-based cosine similarity can be an effective strategy when labeled training data is unavailable.
- BERT Anisotropy is a Consideration: The authors acknowledge the "anisotropy problem" in BERT embeddings, where direct cosine similarity might not always be sufficient for discerning nuanced relatedness. This suggests that while effective, this method has inherent limitations.
Key Results
The paper reports strong performance across the SemRel-2024 task:
- Ranked 1st in MSA (Track B): Achieved a Spearman correlation of 0.49 using the AraBERTv2 model, outperforming the baseline by 0.07 points.
- Ranked 5th in Moroccan (Track A): Achieved a Spearman correlation of 0.83 using the ArBERTv2 model, outperforming the baseline by 0.06 points.
- Ranked 12th in Algerian (Track A): Achieved a Spearman correlation of 0.53.
The augmentation process for Moroccan Arabic led to a performance enhancement. However, for Algerian, neither model surpassed the baseline, and both showed low performance on the development split, suggesting that Algerian data might benefit from similar augmentation.
Limitations and Future Work
- Algerian Data Augmentation: The authors hypothesize that augmenting the Algerian dataset, similar to the Moroccan one, could significantly improve performance.
- Model Anisotropy: The paper acknowledges the potential limitations of direct cosine similarity due to BERT's anisotropy problem, suggesting that more advanced methods might be needed for optimal performance.
- Future Directions: The team plans to explore augmenting more Moroccan and Algerian data using different models. They also intend to experiment with both Arabic mono-dialect and cross-dialect fine-tuning approaches.
Summary updated