Active Slices for Sliced Stein Discrepancy

Wenbo Gong, Kaibo Zhang, Yingzhen Li, José Miguel Hernández-Lobato

Measuring the "distance" between two probability distributions is a cornerstone of modern machine learning. It is essential for everything from evaluating how well a generative model fits data to performing goodness-of-fit (GOF) tests. However, as dimensions increase, standard tools like the Kernelized Stein Discrepancy (KSD) often fall victim to the "curse of dimensionality."

A recent fix is the Sliced Stein Discrepancy (SSD), which projects high-dimensional data onto one-dimensional "slices" to make the problem tractable. But there’s a catch: finding the best slices usually requires slow, iterative gradient-based optimization.

In their ICML 2021 paper, "Active Slices for Sliced Stein Discrepancy," Wenbo Gong and colleagues propose a way to skip the slow optimization entirely using a clever spectral approach.

The Problem: The Cost of Finding the Right Slice

Comparing distributions in high dimensions is notoriously difficult. KSD is a popular tool because it doesn't require knowing the normalization constant of a distribution (which is often impossible to compute). However, in high-dimensional spaces, KSD loses "test power"—it struggles to distinguish between two different distributions because the signal gets lost in the noise.

Sliced variants (SSD and SKSD) solve this by projecting the data onto 1D lines (slices). If you pick the right line, the difference between distributions becomes obvious. The problem is that finding these optimal lines currently relies on gradient descent, which is:

  1. Slow: It requires many iterations.
  2. Unreliable: It often gets stuck in local minima, yielding sub-optimal slices.
  3. Theoretically shaky: It’s hard to prove the discrepancy remains valid if the optimization doesn't reach a global optimum.
Conceptual comparison between optimization-based Sliced Stein Discrepancy and the proposed Active Slices method.

The Solution: Active Slices

The authors tackle this bottleneck in two major steps:

1. Theoretical Relaxation

They prove that as long as you use a "real analytic" kernel, you don't actually need the optimal slice. A finite number of random slices is enough to ensure the discrepancy is mathematically valid (i.e., it equals zero if and only if the distributions are identical). This theoretical guarantee gives researchers the "license" to use sub-optimal but easy-to-find directions without breaking the math.

2. Active Slices via Spectral Decomposition

Since purely random slices might still be weak in practice, the authors propose a way to find highly discriminative slices analytically. Inspired by the Active Subspace Method, they estimate the score difference between the two distributions and perform an eigendecomposition on a matrix derived from these scores.

  • The eigenvectors of this matrix point toward the directions where the distributions differ the most.
  • By picking the top $m$ eigenvectors, they create "Active Slices" that capture the most important information instantly, bypassing the need for iterative training entirely.

Key Results and Performance

The authors evaluated their method on several standard and complex benchmarks, including synthetic GOF benchmarks (Laplace, Multivariate-t, Diffusion), Restricted Boltzmann Machines (RBMs), and Independent Component Analysis (ICA).

Goodness-of-fit test results showing Null Rejection Rate across varying dimensions (top) and training epochs (bottom). Active Slices maintain high performance in high dimensions.
  • Massive Speedups: In goodness-of-fit tests, the Active Slice method was 14x to 80x faster than the state-of-the-art gradient-based SKSD.
  • Better Test Power in High Dimensions: On benchmarks like the Diffusion dataset, Active Slices maintained a high "null rejection rate" (the ability to correctly identify different distributions) even as dimensions increased to 100, while standard KSD and MMD failed completely.
  • Superior Model Learning: In a 150-dimensional ICA task, the Active Slice approach converged significantly faster and reached a much lower Negative Log-Likelihood (NLL) than all other baselines, including "learned" Stein discrepancies.

Limitations and Future Work

While Active Slices offer a massive leap in efficiency, the method does have boundaries:

  • Score Estimation Dependency: The method relies on estimating the score difference. If the sample size is very small or the score estimator is poor, the quality of the Active Slices will degrade.
  • Linearity: The current method focuses on linear projections. Exploring non-linear projections (e.g., using neural networks to define the "slice") could potentially capture even more complex differences.
  • Generalization: The authors suggest that the "Active Slice" philosophy could be applied to other sliced discrepancies, such as the Sliced Wasserstein distance widely used in Optimal Transport, opening the door for broader applications in generative modeling.

Summary updated

Abstract

Sliced Stein discrepancy (SSD) and its kernelized variants have demonstrated promising successes in goodness-of-fit tests and model learning in high dimensions. Despite the theoretical elegance, th...
BibTeX
@inproceedings{gong2021active,
  author = {Wenbo Gong and Kaibo Zhang and Yingzhen Li and José Miguel Hernández-Lobato},
  title = {{Active Slices for Sliced Stein Discrepancy}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=pX0rlQDNB_L},
}
Figure