MultiSeg: Parallel Data and Subword Information for Learning Bilingual Embeddings in Low Resource Scenarios

Efsun Sarioglu Kayi, Vishal Anand, Smaranda MuresanSLTU/CCURL@LREC 2020 · OpenReview

Abstract

Efsun Sarioglu Kayi, Vishal Anand, Smaranda Muresan. Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL).

2020.

Introduction

Cross-lingual word embeddings (CLWEs) are a cornerstone of modern NLP, enabling knowledge transfer from high-resource languages to low-resource ones. However, most standard CLWE models treat words as atomic, "black-box" units. This approach is particularly problematic for morphologically rich languages (e.g., Swahili or Tagalog), where a single root word can take dozens of forms through prefixes and suffixes. In low-resource scenarios, many of these specific word forms appear infrequently, causing standard models to see related strings like "walk," "walking," and "walked" as entirely unrelated. MultiSeg addresses this by leveraging the internal structure of words to share information across related forms, significantly improving data efficiency.

Methodology

MultiSeg builds upon BiSkip (also known as BiVec), a model that learns bilingual embeddings by predicting context words across parallel sentences. The core innovation of MultiSeg is the integration of subword representations directly into the learning objective. Instead of assigning a single vector to a word, a word's vector is calculated as the sum of its subword vectors.

The MultiSeg architecture integrates various subword segmentation strategies—character n-grams, morphological analysis, and BPE—into a unified training pipeline.

The authors experimented with three primary segmentation methods:

  1. Character n-grams: Breaking words into overlapping chunks of 3-6 characters.
  2. Unsupervised Morphological Segmentation: Using "Adaptor Grammars" to identify actual linguistic components like prefixes, stems, and suffixes.
  3. Byte Pair Encoding (BPE): A compression-based subword tokenization method.

Crucially, the model also utilizes stem-level alignment. Rather than just aligning full words between two languages, it aligns the "stems" identified during morphological analysis, providing a cleaner semantic signal for the model to learn equivalents.

Experimental Setup

The study focuses on three genuine low-resource languages—Swahili (swa), Tagalog (tgl), and Somali (som)—using parallel corpora from the IARPA MATERIAL program (ranging from ~24k to ~51k sentences). A simulated low-resource scenario using a 100k sentence subset of German (deu) Europarl data was also included. Models were evaluated using bilingual dictionaries derived from Wiktionary and standard analogy/similarity datasets.

Results and Analysis

MultiSeg consistently outperformed the BiSkip baseline in word translation and syntactic reasoning tasks. In Swahili, the morphological version (MultiSeg_M) achieved a Precision@10 of 0.788 compared to BiSkip’s 0.749.

The gains in syntactic tasks were particularly striking. For Swahili, BiSkip scored only 4.28% on syntactic analogies, while the character n-gram version (MultiSeg_CN) jumped to 30.91%. This suggests that subword information acts as a "cheat code" for syntax; because grammatical functions like tense or plurality are often encoded in suffixes, the model learns these patterns explicitly.

t-SNE visualizations illustrating how MultiSeg clusters morphological variants and their cross-lingual equivalents (e.g., 'done' and its Tagalog forms) more effectively than baseline models.

Qualitative analysis via t-SNE visualizations confirmed that MultiSeg successfully clusters morphological variants (like "done," "did," and "doing") and their cross-lingual counterparts much more tightly than the baseline, which often scatters them across the vector space.

Conclusion and Future Work

MultiSeg demonstrates that morphology serves as a vital bridge in data-starved environments. While the model excelled at syntactic analogies, it showed less improvement in high-level semantic analogies (e.g., "Paris" to "France"). Future work may extend this approach to multi-parallel data, training on several languages from the same family simultaneously to further boost performance in ultra-low-resource settings.

Summary updated

BibTeX
@inproceedings{kayi2020multiseg,
  author = {Efsun Sarioglu Kayi and Vishal Anand and Smaranda Muresan},
  title = {{MultiSeg: Parallel Data and Subword Information for Learning Bilingual Embeddings in Low Resource Scenarios}},
  booktitle = {SLTU/CCURL@LREC},
  year = {2020},
  url = {https://openreview.net/forum?id=JI7o8Povvxs},
}
Figure