Abstract: In this paper, we consider the problem of two-sample hypothesis testing that aims at detecting the difference between two probability densities based on finite samples. The proposed test statistic is constructed by first truncating a sample version of a negative Besov norm and then normalizing it. Here, the negative Besov norm is the norm associated with a Besov space with negative exponent, and is shown to be closely related to a class of commonly used adversarial losses (or integral probability metrics) with smooth discriminators. Theoretically, we characterize the optimal detection boundary of two-sample testing in terms of the dimensionalities and smoothness levels of the underlying densities and the discriminator class defining the adversarial loss. We also show that the proposed approach can simultaneously attain the optimal detection boundary under many common adversarial losses, including those induced by the $\ell_1$, $\ell_2$ distances and Wasserstein distances. Our numerical experiments show that the proposed test procedure tends to exhibit higher power and robustness in difference detection than existing state-of-the-art competitors.
Introduction
Two-sample hypothesis testing is a fundamental task in statistics and machine learning, aimed at determining whether two sets of samples originate from the same underlying probability distribution. This task is critical for various applications, such as detecting data drift in production environments or evaluating the quality of generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs).
A significant challenge in two-sample testing is selecting an appropriate metric to measure the distance between distributions. While traditional metrics like the $\ell_2$ distance are common, they can be overly sensitive to minor perturbations or slight mismatches in data support. Integral Probability Metrics (IPMs), or adversarial losses, offer a more robust alternative but often lack closed-form expressions or are computationally demanding. Maximum Mean Discrepancy (MMD) is a popular IPM, yet its performance is highly sensitive to hyperparameter choices, such as kernel bandwidth.
Methodology: Truncated Negative Besov Norms
This paper introduces a novel framework for two-sample testing based on negative Besov norms. The authors demonstrate that these norms are closely related to a broad class of adversarial losses with smooth discriminators. To make the test computationally feasible, they propose a truncated version of the negative Besov norm, denoted as $T_{\gamma, J}$, which utilizes wavelet transforms to decompose distributional differences across multiple scales.
The test statistic is constructed using U-statistics to provide an unbiased estimator from finite samples. A key step in the procedure is normalization, ensuring that under the null hypothesis, the test statistic follows a standard normal distribution. This allows for straightforward p-value calculation and threshold setting.
Key Insights: Robustness and Optimality
The proposed test is shown to be minimax optimal, meaning it achieves the best possible detection rate for a given sample size. A crucial theoretical insight is that detecting a difference is fundamentally easier than estimating the difference itself, a principle reflected in the test's fast convergence rates.
Furthermore, the test exhibits remarkable robustness to its primary hyperparameter, the truncation level $J$. Unlike MMD, where performance can degrade significantly with a sub-optimal kernel bandwidth, the Besov-norm test maintains high power across a wide range of $J$ values. This makes it particularly suitable for high-dimensional data, such as images, where the underlying data manifold may be complex.
Experimental Results
Numerical experiments on synthetic data and the MNIST dataset validate the theoretical findings. The proposed test consistently outperforms state-of-the-art MMD competitors in terms of power and accuracy.
As shown in the experimental results, the wavelet-based test statistic is closer to a standard normal distribution under the null hypothesis than MMD, facilitating more reliable uncertainty quantification. In practical applications like VAE evaluation on MNIST, the test successfully identified the superior fit of a mixture-of-Gaussians prior over a single Gaussian prior, correctly capturing the dataset's 10-digit structure.
Conclusion
The truncated negative Besov norm provides a powerful and robust framework for two-sample testing. By bridging the gap between theoretical adversarial losses and practical wavelet-based computation, the authors have developed a test that is both mathematically optimal and highly effective in practice. Future work may involve developing adaptive versions of the test that can automatically determine the optimal truncation level based on the data's inherent smoothness.
Summary updated