TL;DR
This paper introduces a new dataset for detecting semantic textual similarity (STS) in Algerian Arabic (ALG), a low-resource language, and its Modern Standard Arabic (MSA) counterpart. They demonstrate that surprisingly simple neural network architectures, specifically two-layer LSTMs, outperform more complex attention-based models on this challenging, noisy data, suggesting that simplicity and robustness are key for under-resourced languages.
What Problem Does This Paper Address?
Semantic Textual Similarity (STS) is the task of determining how semantically related two sentences are, typically by assigning a similarity score. While deep learning models have achieved state-of-the-art results on this task, they have primarily been evaluated on large, well-edited English datasets. This leaves a significant gap in understanding how these models perform on low-resource languages, which often present unique challenges like code-switching, non-standardized orthography, and limited available data. The authors specifically target Algerian Arabic (ALG), a dialect characterized by its complexity and lack of linguistic resources, and its more standardized counterpart, Modern Standard Arabic (MSA). They aim to explore whether neural models can learn to detect semantic equivalence from small, noisy, user-generated datasets, a scenario far removed from the typical large-scale, clean benchmarks.
Methods
The core of the paper involves training and comparing various neural network architectures for binary STS (i.e., determining if sentences are equivalent or not). They employ a siamese network architecture, where two identical networks process each sentence independently, and their representations are then compared. Key components they experiment with include:
- Embeddings: Trainable character or word embeddings, as well as pre-trained word2vec and FastText embeddings.
- Sentence Encoders: Recurrent Neural Networks (RNNs, specifically LSTMs), Convolutional Neural Networks (CNNs), and combinations thereof. They also implement sophisticated attention-based models like Decomposable Attention (DecompAtten) and Enhanced Sequential Inference Model (ESIM).
- Distance/Composition: Standard distance metrics (Euclidean, Manhattan, Cosine) and dense layers for combining sentence representations.
Crucially, the models are trained end-to-end, meaning they learn representations directly from the raw text without relying on pre-defined linguistic features. They also investigate the impact of data augmentation by reversing sentence pairs and the benefit of pre-trained embeddings.
Datasets
The paper introduces a novel ALG STS dataset for detecting semantic textual similarity. This dataset was created by:
- Selecting 1,000 sentences from a larger unlabelled ALG corpus.
- Having two native ALG speakers generate two additional sentences for each: one semantically equivalent and one semantically similar but not equivalent.
- This resulted in 3,000 unique sentence pairs.
- A graded similarity score (0-5) was assigned by three native speakers, followed by manual checking.
- For binary STS, scores 0-4 were merged into "non-equivalent" and score 5 remained "equivalent," yielding 994 equivalent and 2,006 non-equivalent pairs.
They also created a parallel MSA STS dataset by translating the ALG STS dataset into MSA. This MSA dataset has a smaller vocabulary but is more regularized.
Key Insights and Ideas
- Simplicity Trumps Sophistication: The most surprising finding is that relatively simple models, specifically two-layer LSTMs with a trainable embedding layer and a subtraction-based similarity measure, significantly outperform complex attention-based architectures on both ALG and MSA datasets. This suggests that for noisy, low-resource data, robustness and the ability to capture basic semantic relationships are more critical than intricate attention mechanisms.
- Pre-trained Embeddings Can Hurt: Contrary to expectations, using pre-trained word2vec and FastText embeddings on the ALG dataset decreased performance for word-based RNN models. The authors hypothesize this is due to the noise and potential domain mismatch in the unlabelled data used for pre-training.
- Data Augmentation is Beneficial: Augmenting the training data by reversing sentence pairs consistently improved the performance of most models, especially for the minority "equivalent" class, highlighting its importance when dealing with limited data.
- ALG vs. MSA Performance: While MSA is more regular, the performance differences between ALG and MSA were not drastic. However, pre-training embeddings showed a more positive effect on MSA, likely due to its regularity.
Key Results
- The best performing model for binary STS on the ALG dataset was a word-based 2-LSTM encoder with a trainable embedding layer and subtraction as the distance measure, achieving an accuracy of 85.06% on non-augmented data.
- With augmented data, this model improved to 87.48% accuracy.
- Attention-based models like DecompAtten and ESIM performed only slightly better than the baseline, indicating their struggle with the noisy, limited data.
- For MSA, the word-RNN with multiplication performed best on non-augmented data (85.19% accuracy), and with augmentation, the word-RNN with subtraction achieved 88.20% accuracy.
Limitations and Future Work
- Dataset Size: The authors acknowledge that even with augmentation, the dataset size is small, which might be why more complex models struggle.
- Noise and Sparsity: The inherent noise and sparsity in user-generated colloquial data (ALG) pose a significant challenge for learning robust patterns, particularly for CNNs and attention-based models.
- Future Work: The authors propose several avenues for future research:
- Improving the learning capability of neural models on small, noisy datasets without handcrafted features.
- Reducing noise in colloquial data by normalizing spelling variations.
- Further experiments and analyses are needed to fully understand the observed results and their implications for related languages.
Summary updated