TL;DR
This paper introduces ArbEngVec, a novel open-source toolkit providing multiple Arabic-English cross-lingual word embedding models. These models are trained on a massive parallel corpus and evaluated using both intrinsic (word translation) and extrinsic (semantic textual similarity) tasks, offering a valuable resource for NLP research involving these two languages.
What Problem Does This Paper Address?
Natural Language Processing (NLP) tasks have seen significant advancements thanks to word embeddings (WE), which capture semantic properties of words. While mono-lingual WE are well-established, cross-lingual WE (CLWE) are crucial for tasks involving multiple languages, such as machine translation and cross-lingual information retrieval. However, research in CLWE has heavily favored European languages, leaving languages like Arabic relatively under-explored. This paper addresses this gap by developing and evaluating Arabic-English CLWE models, providing a much-needed resource for the NLP community working with these languages.
Methods
The authors propose using two standard shallow neural network architectures for learning word embeddings: CBOW (Continuous Bag-of-Words) and Skip-Gram. These models learn vector representations by predicting a target word from its context (CBOW) or predicting context words from a target word (Skip-Gram).
The novelty lies in their approach to training these models on parallel Arabic-English data. They explore three distinct methods:
- Parallel Mode: This is the most straightforward approach, where sentence pairs are fed directly into the training process without any explicit alignment.
- Word by Word Alignment Mode: This method aligns words within sentence pairs, paying attention to sentence length and using stop-word removal.
- Random Shuffling Mode: Here, bilingual sentence pairs are treated as lists of words, shuffled randomly, and then concatenated to form the training input. This method aims to create a mixed list of Arabic and English tokens.
Datasets
The primary dataset used for training is a large collection of parallel Arabic-English sentences extracted from the Open Parallel Corpus (OPUS). This collection comprises data from various sources, including:
- MultiUN Corpus
- OpenSubtitles
- Tanzil
- News-Commentary
- United Nations (UN)
- Wikipedia
- TED 2013
- GNOME
- Tatoeba
- Global Voices
- KDE4
- Ubuntu
The authors extracted over 93.9 million parallel sentences from this corpus, resulting in approximately 800 million Arabic tokens and 1 billion English tokens.
For evaluation, they used:
- Intrinsic Evaluation: The Word Translation (WT) task, generating 1000 word tuples and finding their k-closest cross-lingual neighbors.
- Extrinsic Evaluation: The Semantic Textual Similarity (STS) task, specifically using the STS2017-Eval10 dataset from SemEval-2017 Task 1 (Cross-lingual Arabic-English).
Key Insights and Ideas
- Random Shuffling Outperforms: The random shuffling mode consistently yielded the best results for both CBOW and Skip-Gram models, suggesting that disrupting the strict sentence alignment can surprisingly improve cross-lingual transfer.
- Skip-Gram Dominance: Across various evaluation metrics and training modes, the Skip-Gram architecture generally outperformed CBOW, particularly in the random shuffling mode.
- Parallel Mode Limitations: The parallel mode, while conceptually simple, showed significantly lower performance, especially in extrinsic evaluations, indicating that direct sentence alignment might not be optimal for capturing cross-lingual semantic relationships due to structural differences between Arabic and English.
- Visualization Reveals Structure: t-SNE visualizations showed that in the parallel mode, Arabic and English words were more distant. However, in the random shuffle mode, translations tended to appear closer together, providing visual evidence of improved cross-lingual mapping.
Key Results
- Intrinsic Evaluation (Word Translation): The random shuffling mode with Skip-Gram achieved the highest accuracy, with Skip-Gram models generally outperforming CBOW across all modes. For instance, in the random shuffle mode, Skip-Gram achieved 93.8% accuracy for the top 10 closest words, while CBOW achieved 90.5%.
- Extrinsic Evaluation (Semantic Textual Similarity): The random shuffling method with Skip-Gram achieved the best correlation rate of 75.7%. This significantly outperformed the parallel mode (18.1% for Skip-Gram) and the word-by-word alignment mode (73.6% for Skip-Gram).
Limitations and Future Work
Limitations:
- The paper notes that the parallel alignment method resulted in very low correlation rates in extrinsic evaluations, suggesting its inadequacy for capturing cross-lingual semantic similarity due to the structural differences between Arabic and English.
- While the random shuffling method showed promising results, the paper acknowledges that it might have "lost the former [monolingual] willingly to stick around the latter [cross-lingual]."
Future Work:
- Integrate the developed models with other classical NLP techniques like word sense disambiguation and named entity recognition to further improve Arabic-English cross-lingual semantic similarity and plagiarism detection.
- Explore better word alignment methods to enhance feature capturing, particularly for the transfer between Semitic (Arabic) and Germanic (English) languages.
- The authors have made their models publicly available as an open-source project (ArbEngVec) on GitHub, encouraging further research and application.
Summary updated