Kernel Conditional Exponential Family

Michael Arbel, Arthur GrettonAISTATS 2018 · OpenReview

Abstract

A nonparametric family of conditional distributions is introduced, which generalizes conditional exponential families using functional parameters in a suitable RKHS. An algorithm is provided for learning the generalized natural parameter, and consistency of the estimator is established in the well specified case. In experiments, the new method generally outperforms a competing approach with consistency guarantees, and is competitive with a deep conditional density model on datasets that exhibit abrupt transitions and heteroscedasticity.

Introduction

In machine learning, understanding the full probability distribution $p(y|x)$ is often more valuable than predicting a single average outcome. This is especially true for multimodal distributions or those exhibiting heteroscedasticity, where noise levels vary with the input. Traditional Conditional Density Estimation (CDE) methods face significant hurdles: Kernel Density Estimation (KDE) scales poorly in high dimensions, and Least-Squares CDE (LS-CDE) can be numerically unstable. While deep learning models like RNADE are powerful, they often require massive datasets and lack theoretical consistency guarantees.

This paper introduces the Kernel Conditional Exponential Family (KCEF), a nonparametric framework that combines the flexibility of Reproducing Kernel Hilbert Spaces (RKHS) with the efficiency of score matching to provide a theoretically consistent and data-efficient estimator.

Methodology

KCEF extends the Kernel Exponential Family to the conditional case by defining the natural parameter as a function of the conditioning variable $x$ within a vector-valued RKHS. This allows the model to smoothly transition the entire shape of the output distribution as the input changes.

Bypassing the Intractable Integral

A major challenge in infinite-dimensional exponential families is the "intractable partition function"—a normalizing constant that is impossible to calculate directly. KCEF overcomes this using score matching. By minimizing the Fisher divergence between the model and the true data, the normalizing constant drops out of the equation, transforming the learning problem into a solvable linear system. To handle higher dimensions, the authors utilize Directed Acyclic Graph (DAG) factorizations to break large joint densities into products of smaller conditionals, reducing computational complexity from $O(n^3 d^3)$ to $O(n^3 d)$.

Experimental Setup

The model was evaluated against LS-CDE and RNADE across several benchmarks:

  • Synthetic 'Grid' Dataset: Used to test scaling with sample size and dimensionality.
  • R Benchmark Datasets: 14 complex, low-dimensional datasets (e.g., geyser, BostonHousing) to test data efficiency.
  • UCI Repository: Challenging datasets like Red Wine and Parkinsons that exhibit non-linear dependencies and abrupt transitions.

Key Results

KCEF demonstrates superior performance in data-limited regimes. On synthetic data, KCEF showed a significantly faster speed of convergence than RNADE as the number of samples increased from 200 to 1,200. It also maintained more robust performance as the data dimensionality increased.

Scaling of log-likelihood with respect to sample size (N) and data dimension (d).

Empirical results on the 14 R benchmark datasets show that KCEF consistently outperforms LS-CDE, often by a significant margin. On larger UCI datasets, KCEF proved highly competitive with deep learning models, specifically emerging as the statistically significant winner on the Parkinsons dataset.

Performance comparison on R benchmark datasets.

Conclusion and Future Work

KCEF provides a powerful nonparametric alternative for CDE, offering both mathematical consistency and strong empirical results on complex, small-to-medium-sized datasets. While its $O(n^3)$ scaling with sample size is a current bottleneck, future research into Nyström-like approximations and automated graph structure discovery could further broaden its applicability to large-scale machine learning problems.

Summary updated

Abstract

A nonparametric family of conditional distributions is introduced, which generalizes conditional exponential families using functional parameters in a suitable RKHS. An algorithm is provided for learning the generalized natural parameter, and consistency of the estimator is established in the well specified case. In experiments, the new method generally outperforms a competing approach with consistency guarantees, and is competitive with a deep conditional density model on datasets that exhibit abrupt transitions and heteroscedasticity.
BibTeX
@inproceedings{arbel2018kernel,
  author = {Michael Arbel and Arthur Gretton},
  title = {{Kernel Conditional Exponential Family}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=VmQWd_ZJti0},
}
Figure