Gaussian Rank Verification

Jeremy Goldwasser, Will Fithian, Giles Hooker

Abstract

Statistical experiments often seek to identify random variables with the largest population means. This inferential task, known as rank verification, has been well-studied on Gaussian data with equal variances. This work provides the first treatment of the unequal variances case, utilizing ideas from the selective inference literature. We design a hypothesis test that verifies the rank of the largest observed value without losing power due to multiple testing corrections. This test is subsequently extended for two procedures: Identifying some number of correctly-ordered Gaussian means, and validating the top-K set. The testing procedures are validated on NHANES survey data.

Introduction

In fields ranging from genetics to recommender systems, researchers frequently aim to identify which variable possesses the highest population mean. Common questions include "Which gene is the best predictor of patient survival?" or "Which product will a user like most?"

The conventional approach involves selecting the variable with the highest observed value (the "winner") and comparing it to the second-highest (the "runner-up"). If the difference is statistically significant, the winner is declared "verified." However, this logic rests on the assumption of homoskedasticity—that all groups share the same variance. In practice, variances are often unequal. A variable with a low mean but high noise can "swing" into first place by chance. Standard tests, which typically only consider the runner-up, fail to account for these "dark horses" lurking further down the rankings, potentially causing false discovery rates to jump from a target 5% to over 34%.

The 'Dark Horse' effect: A high-variance variable (red) can lead to false discoveries if not properly accounted for.

Methodology: Selective Inference for Rank Verification

To address selection bias—the fact that we test the winner specifically because it won—this paper bridges rank verification with selective inference.

The Winner Test

The core contribution is a valid p-value for the null hypothesis that the observed winner is not the true best. This is achieved by conditioning the distribution of the winner’s value on the event of its victory. The authors demonstrate that this conditional distribution follows a truncated normal distribution. The p-value is then calculated based on the tail mass of this distribution, adjusted for the "polyhedral" constraints imposed by the ranking.

Selective inference geometry for testing the winner against the runner-up (a) and third place (b).

Extended Procedures

The framework extends beyond a single winner to two broader procedures:

  1. Ordered Ranking: A sequential test to verify the top $K$ variables in their specific order.
  2. Top-K Sets: A test to verify the "Top-$K$ Set" as a group, regardless of their internal ordering.

Key Insights

  • Comprehensive Comparison: Unlike standard tests that only look at the runner-up, this new test checks the winner against all lower-ranked variables. This is crucial because a high-variance variable ranked 4th or 5th is often a more dangerous competitor than a low-variance runner-up.
  • Efficiency without Penalties: Traditional multiple testing corrections (like Bonferroni) often reduce statistical power. By conditioning on the win within a selective inference framework, the authors maintain validity and power without such penalties.
  • Variance Monotonicity: The test mathematically encodes the intuition that skepticism should increase with the variance of competitors; p-values increase monotonically with the variance of the competitor being compared.

Results and Validation

The methods were validated using both simulations and real-world data from the National Health and Nutrition Examination Survey (NHANES).

Simulation Results

In Gaussian simulations with 5 variables, standard tests exhibited an error rate of 34.4% under heteroskedasticity. In contrast, the proposed Gaussian Rank Verification method strictly maintained the intended 5% error threshold ($\alpha = 0.05$).

NHANES Application

The authors analyzed how education level correlates with various health and economic outcomes:

  • Income: The test confirmed with high confidence that average income increases strictly with every level of educational attainment across all five ranks.
  • Sleep: While college graduates had the highest average sleep, the test could only verify this top rank at a 10% significance level, failing to reliably rank the other groups due to high noise.
  • Mental Health: The test verified that college graduates have the fewest "bad mental health days." It also identified a non-linear trend: the 9th-11th grade group had significantly worse mental health than the 8th-grade-only group.

Conclusion and Future Work

While this framework provides a robust solution for rank verification with unequal variances, it currently assumes independence between variables. Future research is needed to handle correlated variables, which are common in genomic and financial data. Additionally, while the test is designed for Gaussian data, exploring its asymptotic validity for non-Gaussian distributions remains an open area for investigation.

Summary updated

Abstract

Statistical experiments often seek to identify random variables with the largest population means. This inferential task, known as rank verification, has been well-studied on Gaussian data with equal variances. This work provides the first treatment of the unequal variances case, utilizing ideas from the selective inference literature. We design a hypothesis test that verifies the rank of the largest observed value without losing power due to multiple testing corrections. This test is subsequently extended for two procedures: Identifying some number of correctly-ordered Gaussian means, and validating the top-K set. The testing procedures are validated on NHANES survey data.
BibTeX
@inproceedings{goldwasser2025gaussian,
  author = {Jeremy Goldwasser and Will Fithian and Giles Hooker},
  title = {{Gaussian Rank Verification}},
  booktitle = {Stat},
  year = {2025},
  doi = {10.1002/sta4.70087},
}
Figure