A conformal test of linear models via permutation-augmented regressions

Leying GuanOpenReview 2024 · OpenReview

TL;DR

This paper introduces PALMRT, a new conformal hypothesis testing method for linear models that offers robust Type I error control and competitive power, even with complex data structures and non-Gaussian errors. It provides a theoretically guaranteed, assumption-free approach that outperforms existing methods like CPT and RPT in challenging scenarios, making it a practical tool for scientific discovery.

What Problem Does This Paper Address?

When analyzing linear models, especially in fields like biology and medicine, researchers often need to test the significance of individual variables (partial correlation). Traditional methods like the t-test or F-test rely on assumptions about the data's error distribution (e.g., normality) and may not perform reliably with small sample sizes or unusual error structures. While permutation tests offer a more robust alternative by shuffling data to create a null distribution, existing random permutation tests for partial correlation often lack theoretical guarantees for controlling Type I errors (false positives) in a finite-sample, assumption-free manner. This gap can lead to unreliable scientific discoveries. The paper aims to bridge this gap by developing a method that provides these guarantees while maintaining good statistical power.

Methods

The core innovation is PALMRT (Permutation-Augmented Linear Model Regression Test). It's a conformal test that works by augmenting the original regression model with permuted versions of the predictor variables. For each random permutation, PALMRT constructs two test statistics: an "original" statistic based on the actual data and a "permuted" statistic based on the permuted predictors. By comparing these two statistics across many permutations, PALMRT generates a p-value. The key theoretical contribution is that PALMRT guarantees a worst-case Type I error rate of at most $2\alpha$ (where $\alpha$ is the nominal significance level), regardless of the data's design or error distribution, and for any sample size. This is achieved through a novel construction that leverages a specific type of bi-variate function and a transferability property of permutations.

Datasets

The paper uses simulated data for extensive empirical evaluation. It also applies PALMRT to a real-world dataset from a "long-Covid" study investigating cell frequencies in relation to long-Covid status. This dataset was previously analyzed using a t-test, providing a benchmark for comparison.

Key Insights and Ideas

  • Conformal Guarantees for Partial Correlation: PALMRT provides a theoretically sound, assumption-free way to control Type I errors in partial correlation tests, a significant improvement over many existing permutation methods.
  • Robustness to Data Structure: The method's performance is not heavily reliant on specific data designs or error distributions, making it broadly applicable.
  • Power Preservation: Despite its strong theoretical guarantees, PALMRT maintains competitive statistical power, often outperforming other methods like CPT and RPT, especially in challenging scenarios.
  • Practicality: The method is described as robust, practical, and simple to implement, with an R package available.

Key Results

  • Type I Error Control: In simulations, PALMRT consistently controlled Type I error rates below the designated thresholds across various designs and noise distributions, even in challenging settings where other methods like F-test, FLtest, and PERMtest showed inflated errors.
  • Power Comparison: PALMRT demonstrated comparable or superior power to established methods like FLtest, and significantly outperformed CPT and RPT in many scenarios, particularly when dealing with lower signal-to-noise ratios or higher dimensions.
  • Real-World Application: In the long-Covid study, PALMRT successfully validated key findings identified by the t-test, while CPT and RPT showed a drastic loss of power and failed to identify significant biomarkers after multiple corrections.

Limitations and Future Work

  • Paired Design Exception: The paper notes that in specific "paired designs" with many duplicate rows in the covariate matrix Z, CPT can sometimes be more powerful than PALMRT, especially at higher dimensions ($p=15$). This observation prompts a theoretical question about whether PALMRT's power could be enhanced by filtering random permutations to avoid near co-linearity.
  • Future Exploration: The authors suggest exploring this question of filtering permutations for enhanced power in specific designs. They also mention that while PALMRT is generally advantageous for exploratory analyses, achieving very small p-values might still require large sample sizes, a constraint not unique to PALMRT.

Summary updated

Abstract

Permutation tests are widely recognized as robust alternatives to tests based on the normal theory. Random permutation tests have been frequently employed to assess the significance of variables in linear models. Despite their widespread use, existing random permutation tests lack finite-sample and assumption-free guarantees for controlling type I error in partial correlation tests. To address this standing challenge, we develop a conformal test through permutation-augmented regressions, which we refer to as PALMRT. PALMRT not only achieves power competitive with conventional methods but also provides reliable control of type I errors at no more than alpha given any targeted level , for arbitrary fixed-designs and error distributions. We confirmed this through extensive simulations. Compared to the cyclic permutation test (CPT), which also offers theoretical guarantees, PALMRT does not significantly compromise power or set stringent requirements on the sample size, making it suitable for diverse biomedical applications. We further illustrate their differences in a long-Covid study where PALMRT validated key findings previously identified using the t-test, while CPT suffered from a drastic loss of power. We endorse PALMRT as a robust and practical hypothesis test in scientific research for its superior error control, power preservation, and simplicity.
BibTeX
@misc{guan2024conformal,
  author = {Leying Guan},
  title = {{A conformal test of linear models via permutation-augmented regressions}},
  year = {2024},
  url = {https://openreview.net/forum?id=sKybXwqhsy},
  howpublished = {OpenReview},
}
Figure