Active Slices for Sliced Stein Discrepancy

Wenbo Gong, Kaibo Zhang, Yingzhen Li, José Miguel Hernández-Lobato

Abstract

Sliced Stein discrepancy (SSD) and its kernelized variants have demonstrated promising successes in goodness-of-fit tests and model learning in high dimensions. Despite their theoretical elegance, their empirical performance depends crucially on the search of optimal slicing directions to discriminate between two distributions. Unfortunately, previous gradient-based optimisation approaches for this task return sub-optimal results: they are computationally expensive, sensitive to initialization, and they lack theoretical guarantees for convergence. We address these issues in two steps. First, we provide theoretical results stating that the requirement of using optimal slicing directions in the kernelized version of SSD can be relaxed, validating the resulting discrepancy with finite random slicing directions. Second, given that good slicing directions are crucial for practical performance, we propose a fast algorithm for finding such slicing directions based on ideas of active sub-space construction and spectral decomposition. Experiments on goodness-of-fit tests and model learning show that our approach achieves both improved performance and faster convergence. Especially, we demonstrate a 14-80x speed-up in goodness-of-fit tests when comparing with gradient-based alternatives.

Introduction

Comparing probability distributions is a fundamental task in machine learning, essential for goodness-of-fit (GOF) testing and generative model training. While the Kernelized Stein Discrepancy (KSD) is a powerful tool, it often suffers from the "curse of dimensionality," becoming less sensitive as dimensions increase. Sliced variants, such as Sliced SKSD, address this by projecting high-dimensional data onto one-dimensional "slices." However, the effectiveness of these methods depends on finding optimal slicing directions—a task previously reliant on slow, unstable gradient-based optimization prone to local minima.

Methodology: Active Subspace Construction

The proposed Active Slices framework shifts the paradigm from iterative search to analytical construction. The approach is built on two key pillars:

  1. Theoretical Relaxation: The authors prove that the discrepancy remains a valid measure even without strictly "optimal" slices. A finite set of random slices suffices to ensure the discrepancy is zero if and only if the distributions are identical.
  2. Spectral Decomposition: To maximize practical performance, the authors leverage the "active subspace" method. By estimating a sensitivity matrix based on score functions and performing an eigen-decomposition, the top eigenvectors analytically identify the directions where the distributions differ most.
Conceptual framework of Active Slices, illustrating the transition from max-sliced Stein discrepancy to the proposed SKSD with Active Slices via spectral decomposition.

This method supports both Gradient Estimators (GE) and Kernel-smoothed Estimators (KE), allowing it to function even when model score functions are not directly available. A key insight from ablation studies is that "pruning"—retaining only the top few most important slices—often outperforms using a larger, noisier set of directions.

Empirical Results

The method was evaluated on synthetic benchmarks (Laplace, Multivariate-t, Diffusion) and complex tasks like Restricted Boltzmann Machines (RBMs) and Independent Component Analysis (ICA).

Performance and Convergence

  • Massive Speedups: Active Slices achieved a 14x to 80x speedup in GOF tests compared to traditional gradient-based alternatives.
  • Superior Convergence: In model learning tasks (ICA), the Active Slice approach converged significantly faster and reached lower negative log-likelihood (NLL) values than baselines like KSD.
  • Robustness: Unlike iterative methods, Active Slices are insensitive to initialization, making them highly reliable for automated pipelines.
Performance comparison across Laplace, Multi-t, and Diffusion distributions. The top row shows null rejection rates across dimensions, while the bottom row illustrates convergence during training epochs.

Conclusion

Active Slices transform the search for optimal projections in Sliced Stein Discrepancy from a difficult optimization problem into a stable linear algebra task. By combining theoretical validity with significant computational gains, this work provides a robust tool for high-dimensional statistical inference. Future work may explore non-linear generalizations of slice projections and improved score difference estimation methods.

Summary updated

Abstract

Sliced Stein discrepancy (SSD) and its kernelized variants have demonstrated promising successes in goodness-of-fit tests and model learning in high dimensions. Despite their theoretical elegance, their empirical performance depends crucially on the search of optimal slicing directions to discriminate between two distributions. Unfortunately, previous gradient-based optimisation approaches for this task return sub-optimal results: they are computationally expensive, sensitive to initialization, and they lack theoretical guarantees for convergence. We address these issues in two steps. First, we provide theoretical results stating that the requirement of using optimal slicing directions in the kernelized version of SSD can be relaxed, validating the resulting discrepancy with finite random slicing directions. Second, given that good slicing directions are crucial for practical performance, we propose a fast algorithm for finding such slicing directions based on ideas of active sub-space construction and spectral decomposition. Experiments on goodness-of-fit tests and model learning show that our approach achieves both improved performance and faster convergence. Especially, we demonstrate a 14-80x speed-up in goodness-of-fit tests when comparing with gradient-based alternatives.
BibTeX
@inproceedings{gong2021active,
  author = {Wenbo Gong and Kaibo Zhang and Yingzhen Li and José Miguel Hernández-Lobato},
  title = {{Active Slices for Sliced Stein Discrepancy}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=pX0rlQDNB_L},
}
Figure