Abstract
A popular natural language processing task decades ago, word alignment has been dominated until recently by GIZA++, a statistical method based on the 30-year-old IBM models. New methods that outperform GIZA++ primarily rely on large machine translation models, massively multilingual language models, or supervision from GIZA++ alignments itself. We introduce Embedding-Enhanced GIZA++, and outperform GIZA++ without any of the aforementioned factors. Taking advantage of monolingual embedding spaces of source and target language only, we exceed GIZA++’s performance in every tested scenario for three languages pairs. In the lowest-resource setting, we outperform GIZA++ by 8.5, 10.9, and 12 AER for Ro-En, De-En, and En-Fr, respectively.
Introduction
Word alignment—the process of identifying corresponding words between source and target sentences—remains a critical component in various natural language processing (NLP) pipelines. While modern Neural Machine Translation (NMT) systems often bypass explicit alignment, the task is essential for downstream applications such as annotation transfer, dictionary induction, and preserving formatting in translated text.
For over two decades, GIZA++ has served as the industry standard for statistical word alignment. However, GIZA++ relies heavily on word co-occurrence frequencies. In low-resource scenarios or when dealing with rare words (the "long tail" of language), the model lacks sufficient statistical evidence to make accurate predictions. This research introduces Embedding-Enhanced GIZA++ (EE-GIZA++), a method that modernizes the classic tool by incorporating semantic priors from monolingual word embeddings.
Methodology: Bridging Statistics and Semantics
The core innovation of EE-GIZA++ is the integration of cross-lingual word embeddings into the standard Expectation-Maximization (EM) training loop of GIZA++. This process provides a "semantic safety net" for words that appear infrequently in the parallel training corpus.
1. Shared Embedding Spaces
The researchers utilize pre-trained monolingual embeddings (such as fastText) and map them into a shared cross-lingual space using the Procrustes Problem (via VecMap). This allows the system to calculate a similarity score between any source word $x$ and target word $y$, even if they have never appeared together in the training data.
2. Frequency-Based Interpolation
To balance statistical evidence with semantic similarity, the authors introduce a weighting trick. During the update of translation probability tables, the embedding influence is weighted inversely to the word's frequency:
$$score(x, y) = \lambda \frac{p_{map}(y|x)}{freq(x)} + p_{align}(y|x)$$
In this formulation, if a word $x$ is rare ($freq(x)$ is low), the model relies more heavily on the embedding-based mapping ($p_{map}$). Conversely, for high-frequency words, the model prioritizes the statistical alignment probabilities ($p_{align}$) derived from the actual bitext.
Experimental Setup
The model was evaluated on three language pairs with varying resource levels: German-English (De-En), English-French (En-Fr), and Romanian-English (Ro-En). To test the robustness of the approach, the researchers varied the corpus size from a few hundred lines (test set only) to nearly two million lines.
Results and Discussion
EE-GIZA++ demonstrated superior performance over standard GIZA++ across all tested scenarios and language pairs. The gains were most pronounced in extreme low-resource settings.
Key Findings:
- Low-Resource Dominance: In the most constrained settings (aligning only the test set), EE-GIZA++ reduced the Alignment Error Rate (AER) by 10.9 points for De-En and 12.0 points for En-Fr.
- Scalability: Even as the corpus size grew to over a million sentences, the embedding enhancement continued to provide measurable improvements, proving that semantic priors are beneficial even when statistical data is abundant.
- Efficiency: Unlike neural models that require fine-tuning massive Transformers, EE-GIZA++ uses static embeddings, making it a computationally efficient alternative for resource-constrained environments.
Conclusion
EE-GIZA++ successfully modernizes a foundational NLP tool by leveraging the strengths of both statistical modeling and modern vector semantics. By using monolingual data to bolster sparse parallel data, the method provides a robust solution for word alignment in low-resource contexts. Future work will explore integrating these enhancements into more complex IBM models and testing the approach on non-European language pairs.
Summary updated