Abstract
In this paper we show that generative models are competitive with and sometimes superior to discriminative models, when both kinds of models are allowed to learn structures that are optimal for discrimination. In particular, we compare Bayesian Networks and Conditional loglinear models on two NLP tasks. We observe that when the structure of the generative model encodes very strong independence assumptions (a la Naive Bayes), a discriminative model is superior, but when the generative model is allowed to weaken these independence assumptions via learning a more complex structure, it can achieve very similar or better performance than a corresponding discriminative model. In addition, as structure learning for generative models is far more efficient, they may be preferable for some tasks.
Introduction: The Generative vs. Discriminative Debate
In Natural Language Processing (NLP), discriminative models like Logistic Regression or Conditional Random Fields (CRFs) have long been the gold standard. They are praised for their ability to handle overlapping, non-independent features by directly optimizing for classification accuracy. Generative models, conversely, are often dismissed as overly simplistic because they typically rely on rigid independence assumptions—most notably the "Naive Bayes" approach.
This paper challenges the assumption that the weakness of generative models is inherent to the framework itself. By allowing generative models (specifically Bayesian Networks) to automatically learn their own dependency structures from data, they can match or even outperform the best discriminative models while remaining significantly faster to train.
Methodology: Structure Learning in Bayesian Networks
The core technical contribution involves moving away from the standard Naive Bayes structure, where every feature is assumed to be independent given the class label.
Instead, a greedy search algorithm is used to learn the network structure by adding dependencies (edges) between features. This process weakens the independence assumptions, allowing the model to capture complex interactions. A key component of this framework is the use of Witten-Bell smoothing, a sophisticated interpolation technique that helps the model handle rare feature combinations that arise as the structure becomes more complex.
Structure learning in Bayesian Networks is remarkably efficient because parameters can be estimated in "closed form" (via counting). In contrast, discriminative models require iterative optimization every time a new candidate structure is tested, making the development process much slower.
Experimental Setup and Results
The models were evaluated on two classic NLP benchmarks: Prepositional Phrase (PP) Attachment and Semantic Role Labeling (SRL).
Key Findings
- Structure is Everything: When both models use a "Naive" structure, the discriminative model wins. However, once the generative model is allowed to learn dependencies, the gap disappears. In the PP attachment task, the learned Bayesian Network achieved 84.6% accuracy, outperforming the discriminative log-linear model's 83.8%.
- SRL Performance: Results were nearly a tie, with the Bayesian Network hitting 93.5% and the log-linear model slightly ahead at 93.9%.
- Efficiency Gains: Structure search for the Bayesian Network took only 2 hours, compared to 24 hours for the log-linear model. This suggests that generative models are superior candidates for rapid experimentation with complex feature interactions.
- Hybrid Potential: The absolute best performance was often achieved by "Hybrid" models—generative structures where a few hyperparameters are tuned discriminatively on a development set.
Conclusion
This research demonstrates that generative models are highly competitive for NLP classification when freed from the constraints of Naive Bayes. By leveraging efficient structure learning, these models can achieve state-of-the-art accuracy with significantly lower computational costs than traditional discriminative approaches.
Summary updated