When training unsupervised models to uncover the hidden patterns of human language—such as syntactic parse trees, part-of-speech tags, or word alignments—our standard tool is Maximum Likelihood Estimation (MLE). Because we lack human-annotated labels, we optimize a surrogate objective: maximizing the probability of the observed text. The underlying intuition seems elegant: to become highly proficient at predicting surface words, a model must implicitly learn the underlying structural rules of the language.
Yet, decades of research reveal a persistent paradox. As the Expectation-Maximization (EM) algorithm climbs the likelihood curve, the model's actual performance on the target task frequently plateaus or even degrades. A model can achieve a near-perfect statistical fit to a corpus while producing structural representations that are linguistically nonsensical. Understanding why this likelihood-accuracy divergence occurs—and how to circumvent it—is central to building unsupervised systems that align with human intuition.
Why Maximum Likelihood Fails the Structure
The disconnect between data likelihood and structural truth is not a failure of optimization, but a fundamental misalignment of objectives. MLE is designed to compress data and reduce total predictive uncertainty. It does not know, nor does it care, whether its internal representations match human linguistic categories. This misalignment manifests in three primary ways:
1. Model Mis-specification and Independence Assumptions
Generative latent-variable models must rely on simplifying independence assumptions to remain computationally tractable. However, human language routinely violates these assumptions.
As Valentin I. Spitkovsky et al. demonstrated in Viterbi Training Improves Unsupervised Dependency Parsing, when a model's underlying assumptions do not perfectly match the true data-generating process, optimizing for sentence likelihood forces the model to distort its latent structures. The model uses its latent variables as a "buffer" to compensate for its structural flaws, sacrificing grammatical accuracy to maximize surface probability.
2. The Attraction to Statistical Noise
Because MLE rewards any parameter adjustment that explains surface variations, models happily exploit high-frequency statistical noise and idiosyncratic data patterns.
Raphaël Bailly et al. note in Syntax and Geometry of Information that standard likelihood objectives prioritize memorizing specific surface facts over generalizing abstract syntactic rules. For instance, Yong Cheng et al. observe that unsupervised models frequently latch onto highly frequent but structurally irrelevant correlations—like punctuation marks or stop words—because they provide an easy way to explain large portions of the data distribution while ignoring deeper hierarchical logic (Maximum Reconstruction Estimation for Generative Latent-Variable Models).
3. Structural Ambiguity and Simplicity Bias
A surface string of text can often be generated by many different latent configurations. The likelihood objective cannot differentiate between a linguistically valid tree and a degenerate one if both yield the same string probability.
Jinwook Park et al. identified this as a combination of "structural optimization ambiguity" and "simplicity bias" (Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction). Left to its own devices, a model will satisfy the mathematical objective by converging toward repetitive, overly shallow, or trivial structural patterns that minimize structural entropy but fail to reflect human grammar.
Manifestations Across NLP Tasks
This divergence is not isolated to a single algorithm; it is a recurring theme across diverse unsupervised natural language processing tasks.
- Grammar Induction: In unsupervised constituency and dependency parsing, maximizing data likelihood frequently fails to yield linguistically accurate grammars (Lifeng Jin et al., /paper/art_d36281c519ad4465ae35b0f5d56c0b0d; Shay B Cohen et al., /paper/art_ca6daf395fdf4ca9954278f06bca431e. Early work by Franz Beil et al. showed that while lexicalization improves parsing, over-training models on unlexicalized data can degrade their ability to recognize complex linguistic structures even as data likelihood improves (Inside-Outside Estimation of a Lexicalized PCFG for German). Similarly, Elias Ponvert found that maximizing parsing coverage often leads to incorrect category assignments that satisfy technical parser requirements without capturing true grammatical structures (Inducing Combinatory Categorial Grammars with Genetic Algorithms).
- Word Alignment: In machine translation, Robert C. Moore demonstrated that the global maximum of the likelihood function in standard alignment models often corresponds to degenerate parameter settings (Improving IBM Word Alignment Model 1). Specifically, rare words act as "garbage collectors," pooling unrelated target words to artificially inflate sentence likelihood at the expense of alignment accuracy.
- Decipherment: When mapping unknown scripts without parallel data, maximizing the likelihood of the observed ciphertext can cause the model to overfit to the statistical quirks of the target language model rather than learning the true translation mapping, causing accuracy to drift away during late-stage training (Kevin Knight et al., /paper/art_aa8fc37d0c034239b474f21b19b183fb).
Strategies for Aligning Objectives with Accuracy
To bridge the gap between statistical fit and structural truth, researchers have developed techniques that move beyond pure marginal likelihood optimization.
Hard EM and Viterbi Approximations
Standard EM optimizes marginal likelihood by summing over all possible latent structures, which dilutes the discriminative signal by factoring in millions of linguistically implausible trees.
By contrast, Viterbi training (or Hard EM) forces the model to commit to the single most probable latent structure during each training step. As proven by Valentin I. Spitkovsky et al., this heuristic approximation acts as a powerful regularizer (Viterbi Training Improves Unsupervised Dependency Parsing). It prevents the model from hedging its bets across degenerate structures, counterintuitively leading to significantly higher structural accuracy.
Injecting Inductive Biases and Rules
If the data alone cannot disambiguate between valid and invalid structures, the model's search space must be constrained.
- Structural Biases: Christian Clark et al. showed that introducing a simple right-branching structural bias allows neural networks to successfully induce categorial grammars from raw child-directed speech (Categorial grammar induction from raw data).
- Rule Augmentation: Atul Sahay et al. demonstrated that augmenting unsupervised neural parsers with generic, off-the-shelf grammar rules prevents the model from prioritizing data reconstruction over grammatical validity, eliminating syntactically nonsensical trees (Rule Augmented Unsupervised Constituency Parsing).
Alternative Optimization Landscapes
Instead of relying on standard likelihood as the sole metric for model selection and optimization, alternative mathematical formulations can better track structural quality.
Standard MLE: [Raw Text] ---> Maximize Sentence Probability ---> Degenerate Trivial Trees
Alternative Opt: [Raw Text] ---> Maximize Structural Variance ---> Linguistically Valid Trees
(e.g., Surprisal Variance)
For instance, Lifeng Jin et al. introduced Variance of Average Surprisal (VAS) as an alternative predictor for grammar quality (Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction). Because VAS specifically measures how well a grammar distinguishes between content words and function words, it correlates far more tightly with human-like parsing accuracy than data likelihood does.
To navigate these alternative landscapes without getting trapped in poor local optima, Noah A. Smith et al. utilized deterministic annealing techniques (Annealing Techniques For Unsupervised Statistical Language Learning). By gradually morphing a smooth, easily optimizable objective function into the target structural objective, annealing prevents the model from latching onto degenerate parameter settings early in the learning process.
Key Foundational Work
Distributional phrase structure induction (2001) — contributed 2 concept elements to this direction.
Inside-Outside Estimation of a Lexicalized PCFG for German (1999) — contributed 1 concept elements to this direction.