Generative Kernels for Exponential Families

Arvind Agarwal, Hal Daumé IiiAISTATS 2011 · OpenReview

Abstract

This paper introduces "generative kernels," a novel family of closed-form kernels designed for discriminative classifiers (like SVMs) that explicitly encode the generative process of exponential family distributions. By leveraging the natural geometry of the data's statistical manifold, these kernels allow discriminative models to achieve the excellent "small-data" performance typical of generative models, while maintaining superior accuracy as the dataset grows.

Introduction: The Generative-Discriminative Trade-off

In machine learning, a well-known trade-off exists between generative and discriminative models. Generative models, such as Naive Bayes, perform exceptionally well when training data is scarce or when the assumed data distribution is accurate. However, as data volume increases or generative assumptions are violated, discriminative models like Support Vector Machines (SVMs) typically yield superior performance.

Ideally, a hybrid approach would learn efficiently from small datasets while scaling robustly. Previous attempts to bridge this gap, such as Fisher or Heat kernels, are often computationally intractable due to the complexity of calculating the Fisher information metric. This work addresses this bottleneck by proposing a family of mathematically principled kernels with efficient closed-form solutions for exponential family distributions.

Methodology

The authors construct generative kernels using principles from information geometry, focusing on the exponential family (e.g., Gaussian, multinomial, Poisson). The mathematical pipeline involves reframing Maximum Likelihood Estimation (MLE) as a geometric "Bregman median" problem in parameter space. Using Legendre duality, the problem is mapped into a canonical parameter space via Kullback-Leibler (KL) divergence. Symmetrizing this yields the Jensen-Shannon (JS) divergence, whose square root is a negative definite metric. Finally, standard transformations convert this metric into valid positive definite kernels.

Unlike the Fisher kernel, computing these generative kernels only requires evaluating the Legendre dual of the distribution's log-partition function, bypassing intractable integrals while still inducing the Fisher information metric.

Experimental Evaluation

The kernels were evaluated on synthetic and real-world text classification tasks using the multinomial distribution. Real-world tests utilized the WebKB dataset (university webpages) and the Reuters newswire dataset. Synthetic tests varied data density and noise levels to test the limits of the models.

Classification error on the WebKB dataset (student vs. project) as a function of training examples per class.

Key Results and Discussion

Small Data Performance and Scaling

On the WebKB dataset, generative kernels showed a significant relative improvement of approximately 10% over standard kernels when training size was minimal ($n=1$). As shown in the WebKB results, the generative kernels (Exp and Inverse) consistently outperform Naive Bayes (NB) and other baseline kernels as the number of training examples increases.

The "Lower Envelope" Effect

A consistent observation across experiments is the "lower envelope" effect. The error curve of the generative kernel typically tracks the best-performing model at any given data scale—matching Naive Bayes when data is scarce and matching or exceeding standard SVMs as data volume grows. This is particularly evident in synthetic data benchmarks.

Performance comparison on synthetic dense and sparse data, illustrating the lower envelope effect across different data regimes.

Robustness and Geometry

By respecting the statistical manifold, generative kernels provide a more meaningful similarity measure than Euclidean-based kernels. Furthermore, because they are used within discriminative frameworks, they exhibit graceful degradation when generative assumptions are violated, unlike pure generative models which may fail under noise.

Conclusion

Generative kernels successfully bridge the gap between generative and discriminative modeling. They provide SVMs with the efficiency of generative models in small-data regimes while retaining discriminative robustness. Future work includes extending these kernels to unsupervised learning tasks and more complex graphical models.

Summary updated

BibTeX
@inproceedings{agarwal2011generative,
  author = {Arvind Agarwal and Hal Daumé III},
  title = {{Generative Kernels for Exponential Families}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2011},
  url = {https://openreview.net/forum?id=5XvD18mu7b},
}
Figure