Model Invertibility Regularization: Sequence Alignment With or Without Parallel Data

Tomer Levinboim, Ashish Vaswani, David Chiang

Abstract

Traditional sequence alignment models in natural language processing are typically directional, leading to inconsistencies when translating between language pairs in both directions. This paper presents Model Invertibility Regularization (MIR), a novel approach that enforces mathematical invertibility between directional translation models. By regularizing the global parameter space (translation tables) rather than local inferences, MIR improves word alignment and decipherment performance across both parallel and non-parallel datasets.

Introduction

A fundamental challenge in sequence alignment is the discrepancy between directional models. An English-to-French model and a French-to-English model trained independently often produce asymmetric mappings, violating the intuitive property of invertibility. While existing methods like Alignment by Agreement (ABA) attempt to resolve this by forcing agreement on specific sentence pairs, they typically require expensive parallel data and struggle with complex models like IBM Model 4.

Methodology

The proposed Model Invertibility Regularization (MIR) addresses these limitations by imposing a constraint on the translation table ($T$), the internal probability matrix representing the model's "dictionary."

The Invertibility Constraint

For two directional models with translation matrices $T_1$ (Source $\to$ Target) and $T_2$ (Target $\to$ Source), MIR encourages their product to approximate the identity matrix: $T_1 \times T_2 \approx I$. This objective maximizes the "round-trip" probability of a word returning to itself.

Optimization via Concavity

Directly maximizing the trace of the matrix product is computationally difficult. To ensure efficient optimization, the authors apply an entry-wise square root to the translation tables before multiplication. This transformation renders the optimization problem concave, allowing for solution via standard convex programming within a Maximum A Posteriori Expectation-Maximization (MAP-EM) framework.

Experimental Evaluation

The efficacy of MIR was evaluated on two primary tasks:

  1. Word Alignment (Parallel Data): Using Chinese-English (NIST 2009) and Czech-English (News Commentary) corpora.
  2. Decipherment (Non-Parallel Data): Transliterating Japanese Katakana terms back to English using monolingual corpora from Gigaword and Leipzig.

Key Results

MIR demonstrated significant improvements in alignment accuracy. When applied to IBM Model 4, MIR achieved a 72.9 F1 score on Chinese-English alignment, a substantial increase over the 68.4 baseline.

Table 1: Alignment F1 scores for Chinese-English and Czech-English tasks across various models.

Furthermore, MIR showed additive benefits when combined with other methods. The combination of ABA and MIR-HMM resulted in a statistically significant +1.5 BLEU score improvement in translation tasks, as seen in the comparison between baseline EM-HMM and the combined approach.

Table 2: Translation performance (NIST/WMT scores) showing the additive gains of combining ABA and MIR.

In the unsupervised decipherment task, MIR reduced the Whole-Name Error Rate (WNER) from 67% to 59%, effectively closing the performance gap between unsupervised and supervised models by 33%.

Discussion and Conclusion

MIR's primary innovation lies in parameter-level regularization. By enforcing invertibility on the global dictionary, the model naturally produces cleaner, sparser translation tables and more accurate sentence-level alignments. Its compatibility with complex models like IBM Model 4 makes it a versatile tool for various NLP tasks.

Future work will explore extending MIR to account for fertility and distortion parameters, as well as addressing polysemy by encouraging invertibility between word senses rather than surface forms.

Summary updated

Abstract

We present Model Invertibility Regularization (MIR), a method that jointly trains two directional sequence alignment models, one in each direction, and takes into account the invertibility of the alignment task. By coupling the two models through their parameters (as opposed to through their inferences, as in Liang et al.’s Alignment by Agreement (ABA), and Ganchev et al.’s Posterior Regularization (PostCAT)), our method seamlessly extends to all IBMstyle word alignment models as well as to alignment without parallel data. Our proposed algorithm is mathematically sound and inherits convergence guarantees from EM. We evaluate MIR on two tasks: (1) On word alignment, applying MIR on fertility based models we attain higher F-scores than ABA and PostCAT. (2) On Japanese-to-English backtransliteration without parallel data, applied to the decipherment model of Ravi and Knight, MIR learns sparser models that close the gap in whole-name error rate by 33% relative to a model trained on parallel data, and further, beats a previous approach by Mylonakis et al.
BibTeX
@inproceedings{levinboim2015model,
  author = {Tomer Levinboim and Ashish Vaswani and David Chiang},
  title = {{Model Invertibility Regularization: Sequence Alignment With or Without Parallel Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1063},
}
Figure