Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison

Teemu Säilynoja, Paul-Christian Bürkner, Aki VehtariStat. Comput 2022 · OpenReview

TL;DR

This paper introduces a novel graphical test for assessing if data samples come from a specific distribution (or if multiple samples come from the same distribution). It achieves this by visualizing the empirical cumulative distribution function (ECDF) of transformed data with simultaneous confidence bands, offering a more intuitive understanding of deviations than traditional single-value statistics.

What Problem Does This Paper Address?

In computational statistics, a crucial step is evaluating how well a model or a sample fits a hypothesized distribution. This is often done by transforming the data using the Probability Integral Transform (PIT) to see if it follows a uniform distribution. However, existing methods for testing this uniformity, especially with graphical tools, have limitations. They might not provide simultaneous confidence bands that account for the entire ECDF, can be sensitive to binning choices (like histograms), or don't offer insight into the nature of the discrepancy. This paper addresses the need for a robust, intuitive graphical method that provides simultaneous confidence bands for uniformity tests, applicable to both continuous and discrete PIT values, and extends this to comparing multiple samples.

Methods

The core of the paper lies in developing methods to compute simultaneous confidence bands for the ECDF of PIT values.

  • Probability Integral Transform (PIT): This standard technique transforms data from a reference distribution into a uniform distribution. If the data truly comes from the reference distribution, the PIT values should be uniformly distributed.
  • Empirical Cumulative Distribution Function (ECDF): The ECDF of the PIT values is plotted.
  • Simultaneous Confidence Bands: The key innovation is the computation of bands that cover the entire ECDF trajectory with a specified probability (e.g., 95%). This is a significant improvement over pointwise confidence intervals.
  • Simulation-based method: Inspired by existing work, this method involves simulating many ECDF trajectories from a uniform distribution and determining bands that contain a certain proportion of these simulated trajectories.
  • Optimization-based method: A new, more computationally efficient method that uses optimization to find the confidence bands. This method leverages the Markovian property of the ECDF trajectories.
  • Extension to Multiple Samples: The methods are generalized to test if multiple samples originate from the same underlying distribution. This involves a "joint rank transformation" and adapting the confidence band calculations to account for dependencies between samples.

Datasets

The paper uses simulated data to evaluate the performance of their methods. Specific datasets are generated from various distributions (e.g., normal distributions with different means and variances) to test the sensitivity of the methods to deviations from uniformity and to compare multiple samples. Real-world examples are also discussed, particularly in the context of Bayesian inference and MCMC diagnostics (e.g., the "eight schools" model).

Key Insights and Ideas

  • Graphical intuition is powerful: Visualizing the ECDF with simultaneous confidence bands provides a much richer understanding of deviations from uniformity than a single p-value or statistic. It helps identify where and how the data deviates (e.g., in the tails or the center).
  • Simultaneous bands are crucial: Pointwise confidence intervals are insufficient for assessing the entire ECDF. Simultaneous bands ensure that the entire trajectory is considered, preventing false positives that might arise from individual points falling outside pointwise intervals.
  • Optimization offers efficiency: The newly proposed optimization-based method for computing confidence bands is significantly faster than simulation-based approaches, making it more practical for large datasets.
  • PIT for multiple samples is effective: The extension to comparing multiple samples using rank transformations is a valuable tool for diagnostics, especially in MCMC, where it can reveal subtle issues in chain mixing or convergence that other diagnostics might miss.
  • Autocorrelation matters: The paper highlights that autocorrelated samples (common in MCMC) can bias uniformity tests, emphasizing the need for thinning or other methods to address this.

Key Results

  • The proposed graphical test with simultaneous confidence bands demonstrates good performance across various deviation families, often comparable to or better than existing uniformity tests.
  • The optimization-based method for computing confidence bands is substantially faster than simulation-based methods, with speedups of several orders of magnitude reported for larger sample sizes.
  • The multiple sample comparison test effectively detects deviations in means and variances between chains, providing visual insights into convergence issues in MCMC.
  • The method shows sensitivity to autocorrelation in samples, reinforcing the need for pre-processing (like thinning) for accurate results.

Limitations and Future Work

  • Discrete distributions: The current methods are primarily designed for continuous PIT values or discrete PIT values that can be treated as continuous. The paper acknowledges that directly applying the methods to discrete distributions where PIT values are inherently discrete (and can have ties) requires further research, potentially involving randomized PIT values or modifications.
  • Computational cost for many chains: While the optimization method is efficient for single samples, the multiple sample comparison method's computational cost can still scale with the number of chains. The authors suggest pre-computing adjusted values and interpolating for efficiency.
  • LOO-PIT tests: The paper notes that graphical LOO-PIT tests, while useful for visualizing model-data discrepancy, may only approach exact uniformity asymptotically and require further research for precise interpretation.
  • Future work: The authors suggest developing proven and efficient algorithms for handling discrete distributions and further research into the application of graphical LOO-PIT tests.

Summary updated

Abstract

Assessing goodness of fit to a given distribution plays an important role in computational statistics. The probability integral transformation (PIT) can be used to convert the question of whether a given sample originates from a reference distribution into a problem of testing for uniformity. We present new simulation- and optimization-based methods to obtain simultaneous confidence bands for the whole empirical cumulative distribution function (ECDF) of the PIT values under the assumption of uniformity. Simultaneous confidence bands correspond to such confidence intervals at each point that jointly satisfy a desired coverage. These methods can also be applied in cases where the reference distribution is represented only by a finite sample, which is useful, for example, for simulation-based calibration. The confidence bands provide an intuitive ECDF-based graphical test for uniformity, which also provides useful information on the quality of the discrepancy. We further extend the simulation and optimization methods to determine simultaneous confidence bands for testing whether multiple samples come from the same underlying distribution. This multiple sample comparison test is useful, for example, as a complementary diagnostic in multi-chain Markov chain Monte Carlo (MCMC) convergence diagnostics, where most currently used convergence diagnostics provide a single diagnostic value, but do not usually offer insight into the nature of the deviation. We provide numerical experiments to assess the properties of the tests using both simulated and real-world data and give recommendations on their practical application in computational statistics workflows.
BibTeX
@inproceedings{sailynoja2022graphical,
  author = {Teemu Säilynoja and Paul-Christian Bürkner and Aki Vehtari},
  title = {{Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison}},
  booktitle = {Stat. Comput},
  year = {2022},
  url = {https://openreview.net/forum?id=4xmazWoPWk},
}
Figure