Unsupervised Word Alignment Using Frequency Constraint in Posterior Regularized EM

Hidetaka Kamigaito, Taro Watanabe, Hiroya Takamura, Manabu Okumura

Introduction

Word alignment is a fundamental task in statistical machine translation (SMT), identifying correspondences between words in source and target sentences. While traditional generative models like the IBM series are widely used, they are inherently limited by a one-to-many architecture. This restriction struggles to capture the complex many-to-many relationships prevalent in natural language.

A common solution involves training bidirectional models (Source → Target and Target → Source) and enforcing an "agreement" constraint. However, standard agreement treats all words uniformly, which is suboptimal for linguistically distant pairs like Japanese and English. In these cases, function words often lack direct 1-to-1 mappings or are omitted entirely (pro-drop), making strict agreement counterproductive.

Frequency-Based Posterior Regularization

This research introduces a more nuanced approach using the Posterior Regularization (PR) framework. PR allows for the injection of prior linguistic knowledge into the Expectation-Maximization (EM) process without increasing the underlying model's complexity.

The core innovation is the Frequency Constraint, which uses word frequency as a proxy for word type. High-frequency words are categorized as function words (e.g., "the", "is"), while low-frequency words are treated as content words (e.g., "algorithm"). The authors propose three specific constraints:

  • Mismatching Constraint ($f2c$): Penalizes alignments between a function word in one language and a content word in another.
  • Matching Constraints ($f2f$ and $c2c$): Encourages agreement specifically when both directional models identify a function-to-function or content-to-content alignment.
Precision-Recall curves comparing the symmetric baseline with frequency-based constraints (f2f, c2c, f2c) across different datasets.

Experimental Results

The method was evaluated on the French-English Hansard corpus, the Japanese-English Kyoto Free Translation Task (KFTT), and the NTCIR10 patent dataset.

Alignment and Translation Quality

The frequency-based constraints consistently outperformed the standard symmetric baseline. On the KFTT task, the $f2f$ constraint achieved an Alignment Error Rate (AER) of 47.73, a notable improvement over the 48.18 baseline. Furthermore, the NTCIR10 task showed statistically significant gains in BLEU scores, indicating that better word alignment directly translates to higher-quality machine translation.

Impact of constraint parameters on alignment error rate (left) and translation quality (right).

Precision-Recall Trade-off

As shown in the figures above, the proposed constraints improve the precision-recall trade-off. By penalizing "garbage collection"—a phenomenon where rare words are incorrectly mapped to frequent function words just to satisfy model constraints—the model makes more accurate, linguistically sound alignment decisions.

Conclusion

By leveraging word frequency as a simple yet effective feature, this paper demonstrates that unsupervised word alignment can be significantly improved through targeted posterior constraints. This approach is particularly beneficial for language pairs with disparate grammatical structures. Future work may involve more sophisticated word-type identification, such as unsupervised part-of-speech induction, to further refine these constraints.

Summary updated

Abstract

Generative word alignment models, such as IBM Models, are restricted to oneto-many alignment, and cannot explicitly represent many-to-many relationships in a bilingual text. The problem is partially solved either by introducing heuristics or by agreement constraints such that two directional word alignments agree with each other. In this paper, we focus on the posterior regularization framework (Ganchev et al., 2010) that can force two directional word alignment models to agree with each other during training, and propose new constraints that can take into account the difference between function words and content words. Experimental results on French-to-English and Japanese-to-English alignment tasks show statistically significant gains over the previous posterior regularization baseline. We also observed gains in Japanese-toEnglish translation tasks, which prove the effectiveness of our methods under grammatically different language pairs.
BibTeX
@inproceedings{kamigaito2014unsupervised,
  author = {Hidetaka Kamigaito and Taro Watanabe and Hiroya Takamura and Manabu Okumura},
  title = {{Unsupervised Word Alignment Using Frequency Constraint in Posterior Regularized EM}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  doi = {10.3115/v1/d14-1017},
}
Figure