Abstract
We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space (RKHS). Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the development of a flexible sampling strategy that amortizes the cost of Monte-Carlo sampling in the inference stage. The resulting estimator can be easily generalized to kernel conditional exponential families. We establish a connection between kernel exponential family estimation and MMD-GANs, revealing a new perspective for understanding GANs. Compared to score matching-based estimators, the proposed method improves both memory and time efficiency while enjoying stronger statistical properties, such as fully capturing smoothness in its statistical convergence rate while the score matching estimator appears to saturate. Finally, we show that the proposed estimator empirically outperforms state-of-the-art methods.
Introduction
The exponential family of distributions is a cornerstone of statistics and machine learning, encompassing common distributions like the Gaussian, Poisson, and Multinomial. While finite-dimensional exponential families are well-understood, generalizing them to infinite-dimensional spaces using Reproducing Kernel Hilbert Spaces (RKHS) offers a powerful way to model complex, non-parametric data distributions. However, this flexibility comes with a significant mathematical hurdle: the partition function.
In infinite dimensions, the partition function—the normalization constant ensuring the probability distribution integrates to one—becomes an intractable integral. This makes standard Maximum Likelihood Estimation (MLE) impossible. While recent workarounds like score matching avoid the partition function by matching derivatives of the log-density, they introduce their own set of problems, including prohibitive computational costs and slow inference. This paper introduces Doubly Dual Embedding (DDE), a framework that bypasses the partition function while significantly improving efficiency and sampling speed.
The Computational Bottleneck
Existing methods for kernel exponential family estimation primarily rely on score matching. While theoretically sound, score matching requires computing first- and second-order derivatives of the kernel for every dimension and every sample. This leads to a computational complexity of $O(n^3d^3)$ in time and $O(n^2d^2)$ in memory, where $n$ is the number of samples and $d$ is the dimensionality. For high-dimensional data, this scaling is unsustainable.
Furthermore, once a score-matching model is trained, generating new samples is a secondary, expensive task. It typically requires computationally heavy Monte Carlo methods, such as Hamiltonian Monte Carlo (HMC), which must be run from scratch for every new batch of samples.
The Doubly Dual Embedding (DDE) Framework
To address these limitations, DDE reformulates the penalized MLE objective using Fenchel duality. The "doubly dual" approach involves two critical transformations:
- The First Dual (Bypassing the Integral): By applying Fenchel duality to the log-partition function, the authors rewrite the MLE objective as a saddle-point (minimax) optimization problem. This eliminates the need to compute the intractable integral directly.
- The Second Dual (Bypassing Explicit Density): The minimax formulation requires calculating the KL-divergence of a dual distribution, $q$. Normally, this would require knowing the explicit probability density of $q$. By applying Fenchel duality a second time, the authors can parameterize $q$ as a flexible transport mapping—essentially a neural network generator that maps random noise to data samples.
This dual-embedding strategy allows the model to be optimized using Stochastic Gradient Descent (SGD). Because the dual distribution is learned as a generator, the model learns a fast sampler "for free" during the training process.
Theoretical Insights: Bridging to MMD-GANs
A significant theoretical contribution of this work is the discovery of a mathematical bridge between classical statistics and modern deep learning. The authors demonstrate that the saddle-point formulation of penalized MLE is equivalent to a KL-regularized MMD-GAN (Maximum Mean Discrepancy Generative Adversarial Network). This provides a new lens through which to understand GANs, framing them as a form of kernel exponential family estimation.
Additionally, DDE offers superior statistical guarantees. While score matching estimators often saturate at a convergence rate of $O(n^{-2/3})$, DDE fully captures the smoothness of the target distribution, achieving a faster statistical convergence rate of $O(n^{-1})$ under certain conditions.
Empirical Performance
The DDE method was evaluated against state-of-the-art (SOTA) methods on both synthetic and real-world datasets.
Efficiency and Sampling
The most striking result is the improvement in inference speed. On the synthetic two moons dataset, DDE generated samples in just 0.07 seconds, whereas the baseline Kernel Exponential Family (KEF) model required 13.04 seconds using HMC. Beyond speed, the quality of the samples was higher; DDE achieved a Maximum Mean Discrepancy (MMD) score of 0.11, compared to 1.32 for the score-matching baseline.
Density Estimation
In conditional density estimation tasks across 16 real-world benchmark datasets (including geyser, mcycle, and boston highway), DDE outperformed or matched existing methods like KCEF and LSCDE. Notably, DDE achieved these results while using a simpler isotropic kernel, whereas its competitors often required more complex, specialized kernels to remain competitive.
Conclusion
Doubly Dual Embedding represents a significant step forward for kernel exponential families. By transforming a difficult integration problem into a manageable minimax game, DDE reduces the memory bottleneck from $O(n^2d^2)$ to $O(K^2)$ (where $K$ is the number of iterations) and provides an amortized sampler that is orders of magnitude faster than MCMC. While the "doubly dual" formulation introduces more complex optimization dynamics, the resulting gains in efficiency and statistical accuracy make it a powerful tool for non-parametric density estimation.
Summary updated