Classification Logit Two-sample Testing by Neural Networks

Xiuyuan Cheng, Alexander CloningerarXiv 2019 · DOI · OpenReview

Abstract

The recent success of generative adversarial networks and variational learning suggests training a classifier network may work well in addressing the classical two-sample problem. Network-based tests have the computational advantage that the algorithm scales to large samples. This paper proposes a two-sample statistic which is the difference of the logit function, provided by a trained classification neural network, evaluated on the testing set split of the two datasets. Theoretically, we prove the testing power to differentiate two sub-exponential densities given that the network is sufficiently parametrized. When the two densities lie on or near to low-dimensional manifolds embedded in possibly high-dimensional space, the needed network complexity is reduced to only scale with the intrinsic dimensionality. Both the approximation and estimation error analysis are based on a new result of near-manifold integral approximation. In experiments, the proposed method demonstrates better performance than previous network-based tests using classification accuracy as the two-sample statistic, and compares favorably to certain kernel maximum mean discrepancy tests on synthetic datasets and hand-written digit datasets.

Introduction: The Two-Sample Problem in High Dimensions

In statistics, the "two-sample problem" asks a fundamental question: Given two sets of data, do they come from the same underlying distribution? This is a cornerstone of statistical inference, with applications ranging from scientific discovery to validating generative models.

Visualization of two distinct probability densities, p and q, representing the fundamental challenge of the two-sample problem.

Traditional methods, such as the Maximum Mean Discrepancy (MMD) using kernels, have long been the gold standard. However, they face significant hurdles in the modern era of big data. First, the Curse of Dimensionality means that as data dimensionality ($D$) increases, kernel methods often require an exponential amount of data to maintain testing power. Second, Computational Scaling for kernel methods typically scales quadratically ($O(n^2)$) with the number of samples, making them prohibitively slow for massive datasets.

Methodology: The Classification Logit Test

This paper introduces the Classification Logit Test, a novel approach that leverages the expressiveness of neural networks to overcome these limitations. The process fits naturally into a standard machine learning pipeline:

  1. Data Splitting: The two datasets ($X$ and $Y$) are split into training and testing sets.
  2. Training: A binary neural network classifier is trained on the training set to distinguish between $X$ and $Y$ using a standard softmax loss.
  3. The Logit Statistic: Instead of using the final classification accuracy, the authors utilize the logit ($f_\theta$)—the raw, pre-softmax difference between the network's activations for the two classes.
  4. Testing: The test statistic is the difference of the average logits evaluated on the test split.

Mathematically, the logit serves as an estimator for the log-density ratio of the two distributions, acting as a "witness function" that highlights exactly where two distributions differ.

Theoretical Foundations and Error Analysis

A key contribution of this work is the rigorous theoretical guarantee provided for the test's performance. The authors prove that the neural network's required complexity scales with the intrinsic dimensionality ($d$) of the data rather than the ambient dimension ($D$). This explains why neural networks can effectively distinguish between complex image sets that baffle traditional statistical tests.

Decomposition of the total error into estimation and approximation components, illustrating the theoretical bounds of the logit-based test.

The theoretical framework decomposes the total error into approximation error (how well the network can represent the ideal witness function) and estimation error (how well the network learns from finite samples). The authors provide a new result for near-manifold integral approximation to bound these errors, ensuring the test maintains power even in high-dimensional spaces.

Experimental Evaluation

The authors evaluated their method on synthetic benchmarks and real-world datasets:

  • Synthetic Data: Gaussian mixtures and manifold data (e.g., curves in 2D, spheres in 3D).
  • MNIST: Comparing real handwritten digits against those generated by a convolutional autoencoder to test the ability to find subtle discrepancies in high-dimensional image space.

The "net-logit" test consistently outperformed accuracy-based testing ("net-acc"). On the MNIST dataset, the proposed method outperformed various Kernel MMD tests as the sample size increased, demonstrating better stability and faster growth in testing power. Furthermore, the $O(n)$ evaluation complexity makes it significantly more scalable than $O(n^2)$ kernel methods.

Conclusion and Future Work

The Classification Logit Test offers a powerful, scalable alternative to traditional two-sample tests. By focusing on logits rather than accuracy, it captures subtle distributional differences that hard thresholds might miss. While the current theory assumes near-global optimization, future work could explore the impact of specific architectures like ResNets or Transformers and extend the framework to other training objectives like general $f$-divergences.

Summary updated

Abstract

The recent success of generative adversarial networks and variational learning suggests training a classifier network may work well in addressing the classical two-sample problem. Network-based tests have the computational advantage that the algorithm scales to large samples. This paper proposes a two-sample statistic which is the difference of the logit function, provided by a trained classification neural network, evaluated on the testing set split of the two datasets. Theoretically, we prove the testing power to differentiate two sub-exponential densities given that the network is sufficiently parametrized. When the two densities lie on or near to low-dimensional manifolds embedded in possibly high-dimensional space, the needed network complexity is reduced to only scale with the intrinsic dimensionality. Both the approximation and estimation error analysis are based on a new result of near-manifold integral approximation. In experiments, the proposed method demonstrates better performance than previous network-based tests using classification accuracy as the two-sample statistic, and compares favorably to certain kernel maximum mean discrepancy tests on synthetic datasets and hand-written digit datasets.
BibTeX
@misc{cheng2019classification,
  author = {Xiuyuan Cheng and Alexander Cloninger},
  title = {{Classification Logit Two-sample Testing by Neural Networks}},
  year = {2019},
  eprint = {1909.11298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.11298},
}
Figure