Sliced Kernelized Stein Discrepancy

Wenbo Gong, Yingzhen Li, José Miguel Hernández-Lobato

TL;DR

Researchers have developed Sliced Kernelized Stein Discrepancy (maxSKSD), a new way to measure the difference between probability distributions that remains robust even in high dimensions. By projecting complex, high-dimensional data onto optimal one-dimensional "slices," this method significantly improves the power of statistical tests and prevents "mode collapse" when training complex models like Variational Autoencoders (VAEs).

What Problem Does This Paper Address?

In machine learning, we often need to compare two probability distributions—for example, to see if a model's predictions match real-world data (a "goodness-of-fit" test). The Kernelized Stein Discrepancy (KSD) is a popular tool for this because it has a highly desirable property: it only requires the score function (the gradient of the log-density) of the target distribution. This means you don't need to calculate pesky, often intractable, normalization constants.

However, KSD suffers from the "curse of dimensionality." As the number of dimensions increases, the standard kernels (like the RBF kernel) used in KSD become less effective. In statistical tests, this means the test loses "power"—it can no longer reliably tell two different distributions apart. In particle-based inference (like Stein Variational Gradient Descent, or SVGD), it leads to "mode collapse," where the model fails to capture the full diversity of the data because the "repulsive force" that keeps particles apart vanishes in high-dimensional space.

Methods

The authors introduce the Sliced Stein Discrepancy (SSD) and its practical, kernelized variant, maxSKSD. The core idea is inspired by the Radon transform (the mathematics behind CT scans): instead of trying to compare distributions in a massive $D$-dimensional space all at once, you project them onto multiple one-dimensional lines (slices).

Conceptual diagram of the Sliced Stein Discrepancy family

Key technical contributions include:

  • Slicing: They project the score function and test inputs onto 1D directions. This reduces the input to the kernel functions to a single dimension, making them much more robust.
  • Optimal Projections: Rather than picking random directions, they propose finding the optimal slicing directions that maximize the discrepancy, ensuring the test remains highly sensitive to differences.
  • The Kernel Trick: They derive a closed-form solution for these sliced discrepancies using Reproducing Kernel Hilbert Spaces (RKHS), making the method computationally feasible.
  • S-SVGD: They adapt this theory into a new particle inference algorithm called Sliced Stein Variational Gradient Descent. This version uses 1D kernels to maintain a strong repulsive force between particles, even in high dimensions.

Datasets

The authors evaluated their method across several benchmarks:

  • Synthetic Gaussian Benchmarks: Used for goodness-of-fit tests, including "Null" (identical distributions), "Laplace," "Multivariate-t," and "Diffusion" (where distributions differ in only one marginal).
  • Restricted Boltzmann Machines (RBMs): A type of generative graphical model used to test the power of the discrepancy in high-dimensional discrete-like spaces.
  • Binarized MNIST: Used for training Variational Autoencoders (VAEs) to test the S-SVGD algorithm's ability to handle image data.
  • UCI Datasets: Used for training Bayesian Neural Networks (BNNs) for regression tasks.

Key Insights and Ideas

  • 1D is Better than $D$: The "aha moment" of the paper is that while a $D$-dimensional kernel gets "lost" in high-dimensional space, a collection of 1D kernels along the right directions retains its discriminative power.
  • Invertibility Matters: By using the Radon transform framework, the authors ensure that if the sliced discrepancy is zero across all possible slices, the distributions are truly identical. This provides the theoretical "teeth" to the method.
  • Fixing SVGD: The reason SVGD often fails in high dimensions is that the "repulsive force" (which prevents all particles from crowding into one spot) depends on the kernel gradient, which decays to zero in high-D. By slicing, the kernel always sees 1D distances, keeping that repulsive force alive and healthy.

Key Results

  • Superior Test Power: In high-dimensional goodness-of-fit tests (up to 100 dimensions), maxSKSD maintained near-perfect rejection rates for different distributions. In contrast, standard KSD and other baselines saw their performance plummet toward zero as dimensions increased.
Goodness-of-fit test rejection rates across dimensions
  • Faster ICA Training: When training Independent Component Analysis (ICA) models, maxSKSD converged significantly faster than the state-of-the-art "LSD" (Learned Stein Discrepancy) because it has a closed-form solution and doesn't require complex adversarial training.
  • Preventing VAE Posterior Collapse: In VAE training on MNIST, S-SVGD achieved higher label entropy (0.542 vs 0.538 for SVGD) and better accuracy (0.728 vs 0.691), indicating it successfully captured a more diverse and accurate set of latent representations without collapsing.

Limitations and Future Work

  • Optimization Complexity: Finding the optimal slicing direction is a non-convex optimization problem. While the authors used standard gradient descent (Adam) effectively, there is no strict guarantee of finding the global optimum.
  • Computational Overhead: Using many slices increases the computational cost. While more efficient than some adversarial alternatives, it is still more demanding than a standard single-kernel KSD.
  • Deep Kernels: The authors suggest that incorporating "deep kernels" (kernels based on neural network features) into the sliced framework is a promising but currently unexplored avenue.
  • Two-Sample Tests: The authors believe this "slicing" logic could be easily extended to improve Maximum Mean Discrepancy (MMD) for high-dimensional two-sample tests.

Summary updated

Abstract

Kernelized Stein discrepancy (KSD), though being extensively used in goodness-of-fit tests and model learning, suffers from the curse-of-dimensionality. We address this issue by proposing the sliced Stein discrepancy and its scalable and kernelized variants, which employs kernel-based test functions defined on the optimal one-dimensional projections. When applied to goodness-of-fit tests, extensive experiments show the proposed discrepancy significantly outperforms KSD and various baselines in high dimensions. For model learning, we show its advantages by training an independent component analysis when compared with existing Stein discrepancy baselines. We further propose a novel particle inference method called sliced Stein variational gradient descent (S-SVGD) which alleviates the mode-collapse issue of SVGD in training variational autoencoders.
BibTeX
@inproceedings{gong2021sliced,
  author = {Wenbo Gong and Yingzhen Li and José Miguel Hernández-Lobato},
  title = {{Sliced Kernelized Stein Discrepancy}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=t0TaKv0Gx6Z},
}
Figure