TL;DR
This paper tackles the challenge of creating effective English-Malay cross-lingual word embeddings, crucial for leveraging abundant English resources for Malay NLP tasks. They propose augmenting existing bilingual lexicons with machine translation to improve alignment precision, achieving competitive results and demonstrating the impact of lexicon quality on downstream performance.
What Problem Does This Paper Address?
High-quality Malay language resources are scarce, hindering the development of advanced Natural Language Processing (NLP) applications for Malay. Cross-lingual word embeddings offer a solution by enabling the transfer of knowledge from resource-rich languages like English to resource-poor languages like Malay. This allows models trained on English data to be applied to Malay tasks, but requires aligning the semantic spaces of English and Malay word embeddings. Existing methods often rely on large, high-quality bilingual lexicons, which are difficult to obtain for low-resource languages. This paper addresses the challenge of creating accurate English-Malay cross-lingual embeddings by focusing on improving the quality and coverage of the bilingual lexicon used for alignment, particularly for informal text found in social media.
Methods
The core of this work involves aligning pre-trained monolingual word embeddings (English word2vec and Malay word2vec trained on social media data) using an orthogonal transformation approach. The key innovation lies in the augmentation of the initial bilingual lexicon (T-BL) derived from Malaya Documentation. This augmentation involves:
- Lexicon Augmentation: Using machine translation (Google Cloud Translation API and Google Translate) to translate English words from the lexicon into Malay, creating new word pairs.
- Nearest Neighbor Selection: Exploring the impact of including nearest neighbors of seed English words in the augmented lexicon, investigating different numbers of neighbors (1-NN, 5-NN, 10-NN).
- Lexicon Filtering: Implementing filters (WordNet and NLTK-based) to remove noisy or non-English/non-Malay words from the augmented nearest neighbors, aiming to improve the quality of the training lexicon.
- Evaluation Metric: Employing bilingual lexicon induction and measuring performance using P@N (proportion of English words with at least one correct Malay translation among the top N induced translations).
Datasets
- English Monolingual Word Embedding (EWE): Pre-trained on tweets using word2vec (Skip-gram), approximately 3 million words, 400-dimensional vectors.
- Malay Monolingual Word Embedding (MYWE): Pre-trained on tweets and Instagram posts by Husein (2018) using word2vec (Skip-gram), approximately 1.3 million words, normalized and spell-checked. Trimmed to top 200,000 (top200k-MYWE) and top 800,000 (top800k-MYWE) most frequent words.
- Initial Bilingual Lexicon (T-BL): Obtained from Malaya Documentation (Husein, 2018), filtered for invalid and non-English/Malay words. 90% used for training, 10% for testing.
- Gold Standard Bilingual Lexicon (G-BL): Manually created from the test portion of T-BL by extracting Malay translations from the Dewan Bahasa dan Pustaka Malaysia (DBP) dictionary. Contains 1273 entries with 3675 unique Malay words.
- New Bilingual Lexicon (N-BL): Created by augmenting T-BL using machine translation.
Key Insights and Ideas
- Lexicon Quality is Paramount: The study strongly suggests that the quality of the bilingual lexicon is more critical than sheer coverage. A cleaner, more accurate lexicon, even if smaller, leads to better alignment precision.
- Augmentation Benefits, But Noise is a Risk: Augmenting the initial lexicon with machine translation and nearest neighbors can improve performance. However, introducing noisy or irrelevant words can degrade precision, especially beyond a single nearest neighbor.
- 1-NN is Often Optimal: For augmentation, using just one nearest neighbor (1-NN) often strikes a good balance between increased coverage and introducing noise, leading to the best or near-best performance in many scenarios.
- Informal Text Alignment is Possible: The approach successfully aligns embeddings trained on informal social media text, demonstrating its applicability to real-world, noisy language data.
- Machine Translation Tools Matter: Google Cloud Translation API generally yielded better results than Google Translate for augmenting the lexicon.
Key Results
- The proposed method achieved a maximum P@10 of 28.2% on the gold-standard test lexicon (G-BL) using the N-BL lexicon augmented with 1-NN and filtered by WordNet.
- The newly created bilingual lexicon (N-BL) outperformed the baseline T-BL from Malaya Documentation, showing improvements of 1.7% in P@1, 4.2% in P@5, and 2.4% in P@10.
- Augmentation with nearest neighbors showed that precision generally degraded after 1-NN, even with increased coverage.
- The WordNet filter was found to be more effective than the NLTK filter in improving precision.
- The proposed bilingual lexicon (N-BL + 1NN with WordNet filter) outperformed existing bilingual lexicons like MUSE and AN-BL by at least 4.6% in terms of alignment precision, while requiring significantly less computational time.
Limitations and Future Work
- Informal Language Noise: The authors acknowledge that even after filtering, the Malay embeddings still contained significant noise due to being pre-trained on informal corpora. This noise can still affect legitimate word pairings without bilingual supervision.
- Word Frequency Bias: The training and test lexicons were not selected based on word frequency, which the authors consider a strength for generalizability, but it means that very rare words might not be well-represented.
- Word2Vec Choice: The study used word2vec because of the lack of readily available Malay fastText embeddings for social media. The authors suggest pre-training Malay fastText embeddings as a future direction.
- Future Work: The authors plan to evaluate the performance of their English-Malay cross-lingual word embeddings on downstream tasks like emotion classification. They also express interest in exploring semi-supervised and unsupervised approaches for creating cross-lingual embeddings in the future.
Summary updated