Abstract
Traditional sequence alignment models in natural language processing are typically directional, leading to inconsistencies when translating between language pairs in both directions. This paper presents Model Invertibility Regularization (MIR), a novel approach that enforces mathematical invertibility between directional translation models. By regularizing the global parameter space (translation tables) rather than local inferences, MIR improves word alignment and decipherment performance across both parallel and non-parallel datasets.
Introduction
A fundamental challenge in sequence alignment is the discrepancy between directional models. An English-to-French model and a French-to-English model trained independently often produce asymmetric mappings, violating the intuitive property of invertibility. While existing methods like Alignment by Agreement (ABA) attempt to resolve this by forcing agreement on specific sentence pairs, they typically require expensive parallel data and struggle with complex models like IBM Model 4.
Methodology
The proposed Model Invertibility Regularization (MIR) addresses these limitations by imposing a constraint on the translation table ($T$), the internal probability matrix representing the model's "dictionary."
The Invertibility Constraint
For two directional models with translation matrices $T_1$ (Source $\to$ Target) and $T_2$ (Target $\to$ Source), MIR encourages their product to approximate the identity matrix: $T_1 \times T_2 \approx I$. This objective maximizes the "round-trip" probability of a word returning to itself.
Optimization via Concavity
Directly maximizing the trace of the matrix product is computationally difficult. To ensure efficient optimization, the authors apply an entry-wise square root to the translation tables before multiplication. This transformation renders the optimization problem concave, allowing for solution via standard convex programming within a Maximum A Posteriori Expectation-Maximization (MAP-EM) framework.
Experimental Evaluation
The efficacy of MIR was evaluated on two primary tasks:
- Word Alignment (Parallel Data): Using Chinese-English (NIST 2009) and Czech-English (News Commentary) corpora.
- Decipherment (Non-Parallel Data): Transliterating Japanese Katakana terms back to English using monolingual corpora from Gigaword and Leipzig.
Key Results
MIR demonstrated significant improvements in alignment accuracy. When applied to IBM Model 4, MIR achieved a 72.9 F1 score on Chinese-English alignment, a substantial increase over the 68.4 baseline.
Furthermore, MIR showed additive benefits when combined with other methods. The combination of ABA and MIR-HMM resulted in a statistically significant +1.5 BLEU score improvement in translation tasks, as seen in the comparison between baseline EM-HMM and the combined approach.
In the unsupervised decipherment task, MIR reduced the Whole-Name Error Rate (WNER) from 67% to 59%, effectively closing the performance gap between unsupervised and supervised models by 33%.
Discussion and Conclusion
MIR's primary innovation lies in parameter-level regularization. By enforcing invertibility on the global dictionary, the model naturally produces cleaner, sparser translation tables and more accurate sentence-level alignments. Its compatibility with complex models like IBM Model 4 makes it a versatile tool for various NLP tasks.
Future work will explore extending MIR to account for fertility and distortion parameters, as well as addressing polysemy by encouraging invertibility between word senses rather than surface forms.
Summary updated