Abstract
This paper introduces "generative kernels," a novel family of closed-form kernels designed for discriminative classifiers (like SVMs) that explicitly encode the generative process of exponential family distributions. By leveraging the natural geometry of the data's statistical manifold, these kernels allow discriminative models to achieve the excellent "small-data" performance typical of generative models, while maintaining superior accuracy as the dataset grows.
Introduction: The Generative-Discriminative Trade-off
In machine learning, a well-known trade-off exists between generative and discriminative models. Generative models, such as Naive Bayes, perform exceptionally well when training data is scarce or when the assumed data distribution is accurate. However, as data volume increases or generative assumptions are violated, discriminative models like Support Vector Machines (SVMs) typically yield superior performance.
Ideally, a hybrid approach would learn efficiently from small datasets while scaling robustly. Previous attempts to bridge this gap, such as Fisher or Heat kernels, are often computationally intractable due to the complexity of calculating the Fisher information metric. This work addresses this bottleneck by proposing a family of mathematically principled kernels with efficient closed-form solutions for exponential family distributions.
Methodology
The authors construct generative kernels using principles from information geometry, focusing on the exponential family (e.g., Gaussian, multinomial, Poisson). The mathematical pipeline involves reframing Maximum Likelihood Estimation (MLE) as a geometric "Bregman median" problem in parameter space. Using Legendre duality, the problem is mapped into a canonical parameter space via Kullback-Leibler (KL) divergence. Symmetrizing this yields the Jensen-Shannon (JS) divergence, whose square root is a negative definite metric. Finally, standard transformations convert this metric into valid positive definite kernels.
Unlike the Fisher kernel, computing these generative kernels only requires evaluating the Legendre dual of the distribution's log-partition function, bypassing intractable integrals while still inducing the Fisher information metric.
Experimental Evaluation
The kernels were evaluated on synthetic and real-world text classification tasks using the multinomial distribution. Real-world tests utilized the WebKB dataset (university webpages) and the Reuters newswire dataset. Synthetic tests varied data density and noise levels to test the limits of the models.
Key Results and Discussion
Small Data Performance and Scaling
On the WebKB dataset, generative kernels showed a significant relative improvement of approximately 10% over standard kernels when training size was minimal ($n=1$). As shown in the WebKB results, the generative kernels (Exp and Inverse) consistently outperform Naive Bayes (NB) and other baseline kernels as the number of training examples increases.
The "Lower Envelope" Effect
A consistent observation across experiments is the "lower envelope" effect. The error curve of the generative kernel typically tracks the best-performing model at any given data scale—matching Naive Bayes when data is scarce and matching or exceeding standard SVMs as data volume grows. This is particularly evident in synthetic data benchmarks.
Robustness and Geometry
By respecting the statistical manifold, generative kernels provide a more meaningful similarity measure than Euclidean-based kernels. Furthermore, because they are used within discriminative frameworks, they exhibit graceful degradation when generative assumptions are violated, unlike pure generative models which may fail under noise.
Conclusion
Generative kernels successfully bridge the gap between generative and discriminative modeling. They provide SVMs with the efficiency of generative models in small-data regimes while retaining discriminative robustness. Future work includes extending these kernels to unsupervised learning tasks and more complex graphical models.
Summary updated