PMI-Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data

Fatemeh Azadi, Heshaam Faili, Mohammad Javad DoustiACL 2023 · DOI

Introduction

Word alignment is the foundational task of identifying corresponding lexical units between a source sentence and its translation. It serves as a critical component in various natural language processing (NLP) pipelines, including annotation transfer (e.g., part-of-speech tags), bilingual dictionary induction, and the evaluation of machine translation quality.

In the current era of Large Language Models (LLMs), the field has transitioned from traditional statistical models like GIZA++ toward extracting alignments directly from the contextualized embeddings of pre-trained multilingual models such as mBERT or XLM-R. However, most existing extraction methods rely on standard similarity measures like cosine similarity or dot products. This paper identifies a significant limitation in this approach: the hubness problem. In high-dimensional vector spaces, certain "hub" tokens—such as the word "_went" in the authors' examples—exhibit high mathematical similarity to a disproportionately large number of tokens in the target sentence. This phenomenon leads to "greedy" alignments where a single word incorrectly maps to multiple partners, substantially degrading alignment accuracy.

Visualization of the hubness problem (a) and the corrective effect of PMI-based alignment (b).

Methodology

To address the hubness problem, the authors introduce PMI-Align. This method shifts the alignment criterion from raw vector similarity to Point-Wise Mutual Information (PMI), effectively asking how much more likely two tokens are to appear together than would be expected by chance.

The PMI-Align process is executed in three primary stages:

  1. Probability Estimation: The method extracts contextualized embeddings from a pre-trained multilingual language model (specifically utilizing the 8th layer). Cosine similarities between all source and target token pairs are converted into a joint probability distribution $p(x, y)$ via a softmax function.
  2. PMI Calculation: Rather than using raw similarity scores, the method calculates the PMI for each pair: $$PMI(x, y) = \log \frac{p(x, y)}{p(x)p(y)}$$ By normalizing the joint probability $p(x, y)$ by the marginal probabilities $p(x)$ and $p(y)$, PMI-Align naturally penalizes "hubs." If a token is similar to many others (resulting in a high marginal probability), its specific PMI score for any individual pair is reduced.

  3. Alignment Extraction: Finally, a standard "Argmax" approach is employed, intersecting the best source-to-target matches with the best target-to-source matches to generate the final word-level alignments.

Experimental Setup

The researchers evaluated PMI-Align across six diverse public datasets to ensure robust cross-lingual performance: English-Czech (En-Cs), German-English (De-En), English-Persian (En-Fa), English-French (En-Fr), English-Hindi (En-Hi), and Romanian-English (Ro-En). These datasets vary in size from 90 to 2,500 sentences and cover both closely related and linguistically distant language pairs.

Results and Analysis

PMI-Align demonstrated superior performance against established baselines, including SimAlign (Argmax and Itermax variants), Probability Thresholding, and Optimal Transport.

Alignment Error Rate (AER) comparison across different pretrained models and alignment methods.

Key findings from the evaluation include:

  • Improved Accuracy: PMI-Align outperformed all baselines on 5 out of 6 language pairs, achieving an average reduction in Alignment Error Rate (AER) of approximately 1% across different models.
  • State-of-the-Art Benchmarks: When paired with XLM-Align, the method set new benchmarks for unsupervised extraction, reaching an AER of 10.4 on En-Cs and 16.0 on De-En.
  • Effectiveness for Distant Languages: The performance gains were most pronounced for distant language pairs like English-Persian and English-Hindi. In these scenarios, embedding spaces are often less perfectly aligned, making the hubness problem more acute and the PMI regularization more valuable.
  • Zero-Shot Efficiency: The method requires no parallel training data and adds negligible computational overhead (approximately 0.1 seconds per sentence), making it highly suitable for low-resource language applications.

Conclusion and Limitations

PMI-Align offers a simple yet powerful solution to the hubness problem in word alignment by leveraging the statistical properties of PMI. While the method showed slight underperformance on the English-French pair—likely due to the high inherent similarity and subword overlap between the two languages—it represents a significant step forward for unsupervised cross-lingual tasks.

One remaining challenge is the reliance on subword tokenization. As with all LM-based aligners, poor subword splitting in the underlying model can limit alignment quality. Future research may focus on refining probability estimation techniques or extending the PMI framework to other cross-lingual applications such as translation quality estimation.

Summary updated

BibTeX
@inproceedings{azadi2023pmi,
  author = {Fatemeh Azadi and Heshaam Faili and Mohammad Javad Dousti},
  title = {{PMI-Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.782},
}
Figure