Sliced Kernelized Stein Discrepancy

Wenbo Gong, Yingzhen Li, José Miguel Hernández-Lobato

Abstract

Kernelized Stein discrepancy (KSD), though being extensively used in goodness-of-fit tests and model learning, suffers from the curse-of-dimensionality. We address this issue by proposing the sliced Stein discrepancy and its scalable and kernelized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. When applied to goodness-of-fit tests, extensive experiments show the proposed discrepancy significantly outperforms KSD and various baselines in high dimensions. For model learning, we show its advantages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies. We further propose a novel particle inference method called sliced Stein variational gradient descent (S-SVGD) which alleviates the mode-collapse issue of SVGD in training variational autoencoders.

Introduction

Measuring the discrepancy between probability distributions is a fundamental task in statistics and machine learning. Stein Discrepancy (SD) has emerged as a powerful tool because it allows for comparing a sample set to a target distribution $p$ using only its score function $\nabla \log p(x)$. This avoids the need for calculating the normalization constant, making it ideal for unnormalized models like Energy-Based Models.

However, standard Kernelized Stein Discrepancy (KSD) faces a significant challenge: the "curse of dimensionality." As the dimensionality of the data increases, the test power of KSD—its ability to distinguish between different distributions—diminishes rapidly. This limitation also affects particle-based inference methods like Stein Variational Gradient Descent (SVGD), where the repulsive force between particles weakens in high dimensions, leading to "mode collapse."

Methodology: Slicing the Curse

To overcome these limitations, we introduce a "slicing" approach inspired by the Radon Transform. The core idea is to project high-dimensional score functions onto one-dimensional directions (slices) before applying the discrepancy measure.

Conceptual framework of Sliced Stein Discrepancy (SSD) showing the reduction of score and test input dimensions.

The proposed framework consists of three key developments:

  1. Sliced Stein Discrepancy (SSD): A theoretical foundation proving that if the discrepancy is zero across all possible 1D projections, the distributions must be identical.
  2. maxSKSD: Since checking all directions is infeasible, we optimize for the directions that maximize the discrepancy. By applying the kernel trick to these 1D projections, we maintain sensitivity to differences that standard high-dimensional kernels might miss.
  3. S-SVGD: By calculating repulsive forces along these 1D slices, we ensure that the force remains strong even as dimensions grow, effectively preventing particle collapse in high-dimensional spaces.

Experimental Evaluation

We evaluated the proposed maxSKSD and S-SVGD across various benchmarks, including Gaussian synthetic tests, Restricted Boltzmann Machines (RBM), and real-world UCI regression datasets.

Goodness-of-Fit Tests

In high-dimensional goodness-of-fit tests, standard methods like KSD and Maximum Mean Discrepancy (MMD) often fail as the number of dimensions increases. Our experiments demonstrate that maxSKSD maintains high test power even in 100-dimensional spaces.

Goodness-of-fit test results across various distributions. maxSKSD-g (pink) maintains high test power as dimensions increase, unlike standard KSD.

As shown in the figure above, for the "Diffusion" and "Multi-t" benchmarks, the null rejection rate of standard KSD drops toward zero as dimensions increase, while maxSKSD-g remains near 1.0. This confirms that 1D kernels are significantly more robust in high-dimensional settings.

Particle Inference and Model Learning

In Bayesian Neural Network (BNN) tasks and Variational Autoencoder (VAE) training, S-SVGD consistently outperformed standard SVGD. By preventing mode collapse, S-SVGD provided better uncertainty quantification and more diverse sample generation. Furthermore, in training Independent Component Analysis (ICA) models with up to 200 dimensions, maxSKSD achieved faster convergence and higher log-likelihoods than existing baselines.

Conclusion and Future Work

The Sliced Kernelized Stein Discrepancy provides a robust solution to the curse of dimensionality in distribution comparison. By leveraging optimal 1D projections, it maintains high test power and prevents particle collapse in high-dimensional inference. While finding optimal slices introduces some computational overhead, the gains in performance and reliability are substantial. Future work may explore the integration of deep kernels to further enhance the flexibility of the sliced framework.

Summary updated

Abstract

Kernelized Stein discrepancy (KSD), though being extensively used in goodness-of-fit tests and model learning, suffers from the curse-of-dimensionality. We address this issue by proposing the sliced Stein discrepancy and its scalable and kernelized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. When applied to goodness-of-fit tests, extensive experiments show the proposed discrepancy significantly outperforms KSD and various baselines in high dimensions. For model learning, we show its advantages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies. We further propose a novel particle inference method called sliced Stein variational gradient descent (S-SVGD) which alleviates the mode-collapse issue of SVGD in training variational autoencoders.
BibTeX
@inproceedings{gong2021sliced,
  author = {Wenbo Gong and Yingzhen Li and José Miguel Hernández-Lobato},
  title = {{Sliced Kernelized Stein Discrepancy}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=t0TaKv0Gx6Z},
}
Figure