Discussion of "A note on universal inference" by Timmy Tse and Anthony Davison

Mathias Drton, Hongjian Shi, David StriederarXiv 2023 · DOI · arXiv

This paper is a formal discussion of "A note on universal inference" by Timmy Tse and Anthony Davison. It explores how to improve the performance of a relatively new statistical tool called the Split Likelihood Ratio Test (SLRT).

TL;DR

The authors argue that while the Split Likelihood Ratio Test (SLRT) offers a "universal" guarantee for valid inference without standard assumptions, it is often too conservative. They propose that instead of lowering the test's critical values—which risks losing that universal guarantee—researchers should focus on tuning the data-splitting ratio based on the dimensionality of the problem.

What Problem Does This Paper Address?

In classical statistics, many tests (like the standard Likelihood Ratio Test) rely on "regularity conditions"—assumptions that the sample size is large enough or the model is "well-behaved" (e.g., following a normal distribution). When these conditions fail, the tests can become invalid.

Universal Inference, introduced recently by Wasserman et al. (2020), provides a workaround called the SLRT. It is valid for any sample size and requires almost no assumptions. However, there is a "price of universality": the test is often highly conservative, meaning it lacks "power" (the ability to correctly detect a true effect).

The authors address the debate on how to make the SLRT more powerful without breaking the very "universality" that makes it special.

Methods

The paper compares two different philosophies for improving the SLRT:

  1. Adjusting Critical Values (Tse and Davison): This approach suggests using a smaller "threshold" for rejecting the null hypothesis. This is justified by exploiting the independence between nuisance parameters (parameters we don't care about) and the parameters of interest in specific models.
  2. Tuning the Split Ratio (Drton, Shi, and Strieder): The SLRT works by splitting data into two parts: one to estimate the parameters and one to test them. The authors of this paper argue for changing the ratio ($\gamma$) of how the data is split (e.g., 50/50 vs. 90/10) rather than changing the threshold.

The authors use large-sample theory to derive the limiting distributions of the SLRT under both the null hypothesis and "local alternatives" (scenarios where the null is slightly false).

Datasets

This is a theoretical discussion paper and does not use real-world datasets. Instead, it utilizes:

  • Mathematical Models: Specifically, the "normal location model" and "Gaussian mixture models."
  • Simulations: The authors present simulations of a high-dimensional scenario (testing a 45-dimensional hypothesis within a 50-dimensional parameter space) to demonstrate how different split ratios affect power and size.

Key Insights and Ideas

  • The Universality Trade-off: Lowering critical values (the Tse and Davison approach) increases power in "regular" cases but loses the "universal guarantee." In "irregular" or complex problems, a lowered threshold might lead to a high rate of false positives (Type I errors).
  • Split Ratios are Dimension-Dependent: Previous work suggested a split ratio of $\gamma = 2/3$ was generally best. This paper reveals that this is only true for low-dimensional problems.
  • High-Dimensional Advantage: In high-dimensional settings, the authors show that a smaller split ratio (using less data for estimation and more for testing, e.g., $\gamma = 1/10$) actually yields much higher power.
  • Safety First: Tuning the split ratio is a "safer" optimization. Because it doesn't change the fundamental logic of the test, the SLRT remains universally valid in finite samples, regardless of whether the problem is "regular" or not.

Key Results

  • Power vs. $\delta$: In Figure 1(a), the authors show that for a 45-dimensional hypothesis, the power of the test increases significantly as the split ratio $\gamma$ decreases. A ratio of $1/10$ (purple line) outperforms $9/10$ (red line) by a wide margin.
  • Size Control: Figure 1(b) demonstrates that the "true size" (the probability of a false positive) stays well below the nominal significance level $\alpha$, confirming that the test remains valid (if conservative) across all split ratios.

Limitations and Future Work

  • Limited Power Gains: The authors acknowledge that tuning the split ratio has a "more limited impact on power" compared to the more aggressive approach of changing critical values.
  • Complexity of Choice: There is no single "perfect" split ratio; the optimal $\gamma$ depends on the dimensionality of both the null and alternative hypotheses.
  • Future Direction: The authors point toward their more extensive work (Strieder and Drton, 2022) for a detailed framework on how to choose the optimal splitting ratio based on specific model dimensions.

Summary updated

Abstract

Invited discussion for Stat of "A note on universal inference" by Timmy Tse and Anthony Davison (2022)
BibTeX
@misc{drton2023discussion,
  author = {Mathias Drton and Hongjian Shi and David Strieder},
  title = {{Discussion of "A note on universal inference" by Timmy Tse and Anthony Davison}},
  year = {2023},
  eprint = {2303.17689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.17689},
}
Figure