Maximum Reconstruction Estimation for Generative Latent-Variable Models

Yong Cheng, Yang Liu, Wei Xu

Introduction

Unsupervised natural language processing often relies on generative latent-variable models, such as Hidden Markov Models (HMMs), to induce structures like part-of-speech (POS) tags or word alignments. Traditionally, these models are trained using Maximum Likelihood Estimation (MLE). However, MLE frequently suffers from a "laziness" problem, where it captures high-frequency but irrelevant correlations—such as punctuation or stop words—rather than the meaningful syntactic or semantic structures of interest.

In this work, Cheng et al. (2017) propose Maximum Reconstruction Estimation (MRE) as a robust alternative. By shifting the objective from explaining observed data to reconstructing it through a latent bottleneck using shared parameters, MRE encourages the discovery of more accurate latent structures. This approach addresses the fundamental limitation of MLE by forcing the model to focus on the underlying structural dependencies that define a language's syntax or translation logic.

Problem Statement

Generative models characterize the joint probability $P(x, z)$ of observed data $x$ and latent structure $z$. Standard MLE maximizes the marginal probability $P(x) = \sum_z P(x, z)$. In language tasks, this often leads the model to over-fit to statistical noise. For instance, in POS induction, a model might associate a latent state with commas simply because they are ubiquitous, failing to capture grammatical categories like nouns or verbs. The challenge is to find a training objective that ignores this noise and focuses on the underlying structural dependencies.

Methodology

MRE adopts an encoding-reconstruction framework. Instead of maximizing $P(x)$, the objective is to maximize the probability of reconstructing $x$ from itself via a latent variable $z$: $$P(\hat{x}|x; \theta) = \sum_z P(z|x; \theta) P(\hat{x}|z; \theta)$$

A critical constraint in MRE is parameter sharing: both the encoding step $P(z|x)$ and the reconstruction step $P(x|z)$ utilize the same set of model parameters $\theta$. This forces the generative model to learn representations that are simultaneously useful for both inference and generation, preventing the "cheating" that can occur when encoder and decoder parameters are independent. The authors derive exact gradients for MRE, showing it is computationally feasible for HMMs and IBM translation models using efficient dynamic programming. Specifically, the MRE gradients involve a distribution $Q$ that weights paths by the square of their probabilities, effectively sharpening the model's focus on the most probable latent structures.

Experimental Setup

The effectiveness of MRE was evaluated on two unsupervised tasks:

  1. POS Induction: Using the English Penn Treebank (46K sentences) to cluster words into syntactic categories without labeled examples.
  2. Word Alignment: Using the FBIS Chinese-English corpus (240K parallel sentences) to identify word correspondences.

Results and Discussion

MRE demonstrated superior performance across both tasks. In POS induction, MRE consistently outperformed MLE as the number of latent states increased. For a 50-state HMM, MRE achieved an accuracy of 67.39%, significantly higher than the 61.08% achieved by MLE. Beyond accuracy, MRE also showed significant improvements in Variation of Information (VI), a metric used to evaluate clustering quality. Lower VI values indicate that the induced clusters are more consistent with the ground truth syntactic categories.

Performance comparison of MLE and MRE for unsupervised POS induction across different state counts.

Qualitatively, MRE proves more robust to frequency bias. Under MLE, latent states often become "garbage collectors" for high-frequency tokens like commas. In contrast, MRE focuses on words with stronger structural roles. For example, a latent state corresponding to verbs might be dominated by punctuation under MLE, but correctly identifies words like "said" and "reported" under MRE. This reconstruction bottleneck ensures that for a word to be successfully passed through a latent tag and reconstructed, that tag must capture essential information about the word's identity and context.

Comparison of top words associated with a latent state under MLE vs. MRE, illustrating MRE's robustness to high-frequency noise.

For word alignment using IBM Model 2, MRE reduced the Alignment Error Rate (AER) from 40.28 to 38.33. Furthermore, MRE is computationally efficient; while more intensive than standard Expectation-Maximization (EM), it remains orders of magnitude faster than discriminative alternatives like CRF Autoencoders, which can take weeks to train on similar hardware.

Conclusion

While MRE shows great promise for HMMs and IBM Model 2, future work includes extending the framework to more complex structures like Probabilistic Context-Free Grammars (PCFGs) and more sophisticated translation models like IBM Model 4. Additionally, investigating the impact of different initialization strategies remains an open area for research, as unsupervised methods are often sensitive to their starting points.

Summary updated

Abstract

Generative latent-variable models are important for natural language processing due to their capability of providing compact representations of data. As conventional maximum likelihood estimation (MLE) is prone to focus on explaining irrelevant but common correlations in data, we apply maximum reconstruction estimation (MRE) to learning generative latent-variable models alternatively, which aims to find model parameters that maximize the probability of reconstructing the observed data. We develop tractable algorithms to directly learn hidden Markov models and IBM translation models using the MRE criterion, without the need to introduce a separate reconstruction model to facilitate efficient inference. Experiments on unsupervised part-of-speech induction and unsupervised word alignment show that our approach enables generative latent-variable models to better discover intended correlations in data and outperforms maximum likelihood estimators significantly.
BibTeX
@inproceedings{cheng2017maximum,
  author = {Yong Cheng and Yang Liu and Wei Xu},
  title = {{Maximum Reconstruction Estimation for Generative Latent-Variable Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10971},
}
Figure