SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings

Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze

Abstract

Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

Word alignment—the task of identifying corresponding words between source and target sentences—is a cornerstone of machine translation and cross-lingual NLP. Traditionally, high-quality alignment has relied on statistical models like GIZA++ or eflomal, which require massive parallel corpora to learn mappings. SimAlign introduces a paradigm shift by leveraging pre-trained multilingual word embeddings (e.g., mBERT) to achieve state-of-the-art alignment without any parallel training data.

The Challenge of Parallel Data

Parallel data is a scarce resource for the vast majority of the world's language pairs. While abundant for English-French, it is nearly non-existent for low-resource languages or specialized domains like legal and medical text. SimAlign addresses this by asking whether the cross-lingual knowledge already captured in multilingual models can be used for zero-shot word alignment.

Methodology

SimAlign treats alignment as a similarity problem within a shared multilingual embedding space. The process involves generating a similarity matrix between source and target words and extracting alignments using one of three algorithms:

  1. Argmax: A baseline identifying mutual nearest neighbors.
  2. Itermax: A novel iterative approach that progressively discounts similarities of already-aligned words to handle one-to-many mappings.
  3. Match: A global optimization using the Hungarian algorithm for maximum-weight bipartite matching.
Overview of the SimAlign pipeline and subword-to-word alignment conversion.

A critical component of this methodology is handling subword units. Since models like mBERT use subword tokenization, SimAlign aligns at the subword level and then aggregates these into word-level alignments using specific heuristics, as shown in the pipeline overview above.

Experimental Setup

The researchers evaluated SimAlign across six language pairs (Czech, German, Persian, French, Hindi, and Romanian) paired with English. They compared static embeddings (fastText via VecMap) against contextualized embeddings (mBERT, XLM-R). Baselines included statistical aligners trained on up to 1.9 million parallel sentences.

Key Insights

  • The "Layer 8" Sweet Spot: Middle layers of mBERT (specifically layer 8) provide the best balance of context and general semantic information for alignment.
  • Contextual Superiority: Contextualized embeddings significantly outperform static ones by resolving polysemy through surrounding context.
  • Subword Importance: Aligning subwords is essential for morphologically complex languages and rare words, allowing the model to match roots and suffixes.
  • Zero-Resource Performance: Remarkably, SimAlign with zero parallel data can outperform statistical aligners trained on 100,000 sentences.

Results and Robustness

SimAlign consistently outperformed traditional methods in low-resource settings. On English-German, it achieved an $F_1$ score 5 percentage points higher than eflomal (trained on 100k sentences).

Examples of word alignments across diverse language pairs and mixed-language sentences.

As illustrated in the examples above, SimAlign is robust across distant language pairs (e.g., German-Uzbek) and even handles code-switching or mixed-language sentences where traditional statistical models often fail.

Conclusion and Future Work

SimAlign demonstrates that high-quality word alignment is possible without parallel training data, making it a vital tool for low-resource NLP. While currently limited to the languages supported by pre-trained models, future work aims to improve fertility modeling and explore hybrid approaches that incorporate parallel data when available to further enhance performance.

Summary updated

BibTeX
@inproceedings{sabet2020simalign,
  author = {Masoud Jalili Sabet and Philipp Dufter and François Yvon and Hinrich Schütze},
  title = {{SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=GMrDiTLYrjG},
}
Figure