An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators

Percy Liang, Michael I. Jordan

Introduction

In machine learning, practitioners often face a choice in how to train a model for a prediction task (predicting $y$ from $x$). Should we model the joint distribution $p(x, y)$ (Generative), the conditional distribution $p(y|x)$ (Discriminative), or use a computationally cheaper approximation like the product of local conditionals (Pseudolikelihood)?

While empirical wisdom suggests discriminative models usually perform better with enough data, and generative models perform better with very little data, a rigorous comparison within a single model family has historically been lacking. Standard learning theory often provides "upper bounds" on error that are too loose to distinguish between these estimators. This paper provides a unified theoretical framework to compare these three heavyweights of statistical learning. By using asymptotic analysis—examining how estimators behave as the number of data points $n$ approaches infinity—the authors prove a fundamental trade-off: modeling more of the data (generative) reduces variance and requires less data to converge, but it makes the model "fragile" and highly sensitive to being wrong about the underlying data distribution (misspecification).

Theoretical Framework

The authors treat all three approaches as special cases of Composite Likelihood Estimators within the Exponential Family. They define an "equivalence function" that partitions the data space:

  • Generative models use a single partition (the whole space).
  • Discriminative models partition the space based on the input $x$.
  • Pseudolikelihood models use even finer, local partitions.

Using Taylor expansions and the "Delta method," the authors derive the asymptotic variance of the parameters and the expected prediction error (log-loss). They decompose the total prediction error into approximation error (how far the best model in our family is from the truth) and estimation error (the noise introduced by having finite data).

Key Insights: The Variance-Bias Trade-off

A central finding of the paper is the formalization of the variance-bias trade-off among these estimators. Modeling the marginal distribution of the input $x$ (as generative models do) acts as a regularizer. It provides more "statistical strength," which reduces the variance of the parameter estimates. However, if the model's assumptions about $x$ are wrong, this strength pulls the model toward the wrong solution.

Asymptotic variance and total error comparisons across generative, fully discriminative, and pseudo-discriminative estimators under varying conditions.

The asymptotic analysis reveals distinct convergence rates:

  • Well-specified models: When the true distribution is in our model family, all estimators converge to the best parameters at a rate of $O(n^{-1})$. Furthermore, "coarser" partitions (modeling more dependencies, like joint likelihood) generally lead to more efficient estimators.
  • Misspecified models: When assumptions fail, discriminative models still converge at $O(n^{-1})$, but generative and pseudolikelihood models slow down significantly to $O(n^{-1/2})$.

Discriminative models are naturally shielded from errors in modeling the input distribution $p(x)$ because they ignore it entirely, focusing only on the mapping $x \to y$.

Empirical Validation

The paper validates its theoretical findings using two primary setups: a synthetic four-node binary Markov Random Field (MRF) to precisely control misspecification, and a real-world Part-of-Speech (POS) tagging task using the Wall Street Journal (WSJ) portion of the Penn Treebank.

In the well-specified synthetic MRF experiments, the generative estimator achieved the lowest error, confirming its superior statistical efficiency when assumptions hold. However, real-world data is rarely perfectly specified.

Performance on real, misspecified data (WSJ POS tagging), showing the discriminative model outperforming the generative approach.

On the WSJ dataset, the discriminative estimator (a Conditional Random Field) significantly outperformed the generative one (a Hidden Markov Model). The HMM's test log-loss was 4.945 compared to the CRF's 3.120.

To perfectly illustrate that the generative model's weakness isn't its underlying math but its assumptions, the authors performed a "flip" experiment. When they took the HMM trained on real data and used it to generate synthetic data (meaning the generative model's assumptions were now perfectly correct for the dataset), the generative model suddenly became the winner, achieving a test accuracy of 0.898 versus the discriminative model's 0.879.

Limitations and Future Work

While comprehensive, the analysis is strictly tied to regular exponential families. This broad category doesn't cover all modern deep learning architectures, though many components behave similarly. Additionally, asymptotic analysis describes behavior with "large" $n$ and may not perfectly predict outcomes in extremely low-data regimes where a generative model’s variance reduction might outweigh a large bias. Future directions suggested by the authors include extending this analysis to latent variable models (marginal likelihoods) and "curved" exponential families to encompass a wider variety of directed graphical models.

Summary updated

Abstract

Statistical and computational concerns have motivated parameter estimators based on various forms of likelihood, e.g., joint, conditional, and pseudolikelihood. In this paper, we present a unified framework for studying these estimators, which allows us to compare their relative (statistical) efficiencies. Our asymptotic analysis suggests that modeling more of the data tends to reduce variance, but at the cost of being more sensitive to model misspecification. We present experiments validating our analysis.
BibTeX
@inproceedings{liang2008asymptotic,
  author = {Percy Liang and Michael I. Jordan},
  title = {{An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators}},
  booktitle = {International Conference on Machine Learning},
  year = {2008},
  doi = {10.1145/1390156.1390230},
}
Figure