Finding Distributions that Differ, with False Discovery Rate Control

Yonghoon Lee, Edgar Dobriban, Eric Tchetgen TchetgenarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a new statistical framework for identifying which groups in a dataset differ from a reference distribution while strictly controlling the False Discovery Rate (FDR). By inventing "batch conformal p-values," the authors provide a distribution-free method that allows researchers to reuse reference data across multiple comparisons without compromising statistical validity.

What Problem Does This Paper Address?

In fields like medicine or economics, researchers often need to compare a "control" group (the reference) against many different "treatment" or "sub-population" groups. The goal is to pinpoint exactly which groups deviate from the norm.

This presents two major challenges:

  1. The Multiple Testing Problem: If you run 100 tests, you're bound to find "significant" differences just by chance. Controlling the False Discovery Rate (FDR)—the expected proportion of false discoveries among all discoveries—is essential.
  2. Data Scarcity and Dependency: Standard methods often require splitting the reference data into $K$ separate pieces to keep tests independent. If your reference group is small, this destroys your statistical power. However, if you reuse the same reference data for every comparison, the tests become dependent, which usually breaks the math behind standard FDR-control tools like the Benjamini-Hochberg (BH) procedure.

Methods

The authors bridge this gap using Batch Conformal Inference.

  • Batch Conformal P-values: Traditional conformal prediction generates a p-value for a single new data point to see if it's an outlier. The authors extend this to "batches" (entire groups). They calculate a p-value based on how a specific quantile (e.g., the median or the 90th percentile) of a comparison group ranks against the reference group.
  • Score Functions: The method is flexible; you can use simple values (like blood pressure) or complex "nonconformity scores" (like residuals from a machine learning model) to define what "different" looks like.
  • Theoretical Breakthrough: The core contribution is a mathematical proof showing that these batch p-values satisfy a property called Positive Regression Dependence (PRDS). This is a specific type of "good" dependency. Because they proved PRDS holds even when reusing reference data, they can safely apply the standard Benjamini-Hochberg procedure to guarantee exact FDR control.
  • Inductive Rank Construction: To prove the PRDS property, the authors developed a novel inductive technique for constructing rank vectors, which may have broader applications in probability theory.

Datasets

The paper validates the method using three distinct approaches:

  1. Synthetic Simulations: Testing 1D and multivariate shifts under both normal and "heavy-tailed" (Cauchy) distributions to compare power against "oracle" tests (tests that know the true underlying distribution).
  2. HALT-C Study: A medical dataset tracking Hepatitis C treatment effects. The authors used it to identify specific patient demographics (categorized by age, BMI, and sex) that showed significantly different treatment responses in platelet counts.
  3. Current Population Survey (CPS): A massive US Census dataset. The authors identified specific sub-populations (defined by education, race, and age) that work significantly longer hours than a reference group.

Key Insights and Ideas

  • Reusing Data Safely: The biggest "aha moment" is that you don't have to sacrifice power by splitting your reference set. You can "double-dip" into your reference data for multiple tests as long as you use the batch conformal framework.
  • Targeting the Tails: Unlike the popular Wilcoxon rank-sum test, which looks at the "bulk" of the distribution, this method allows researchers to target specific quantiles. For example, you can specifically look for treatments that help the sickest 10% of patients, even if the average patient stays the same.
  • Distribution-Free Power: In simulations, this method performed nearly as well as "oracle" tests that had perfect knowledge of the data's distribution, proving that you don't need to assume "normality" to get high-quality results.

Key Results

  • Guaranteed FDR Control: The method consistently controlled the FDR at the target level (e.g., 5% or 10%) across all simulations, regardless of how many groups were tested.
  • Superior Power: It significantly outperformed "subsampling" approaches (which only use one data point per group) and was more robust than t-tests when the data was non-normal.
  • Computational Efficiency: Unlike permutation tests, which can be computationally "expensive" because they require thousands of reshuffles, the batch conformal p-value is calculated using a closed-form combinatorial formula, making it much faster for large-scale data.

Limitations and Future Work

  • Hyperparameter Selection: The user must choose which quantile ($\eta_k$) to test. While the median is a safe default, the "best" quantile depends on the specific scientific question, and the paper suggests data-driven ways to tune this.
  • Independent Comparison Groups: The current proof assumes that while groups share a reference, they are independent of each other. Future work could explore settings where comparison groups also share data or have internal dependencies.
  • Pairwise Comparisons: The current framework is "one-vs-many." Extending this to "many-vs-many" (comparing every group to every other group) remains an open challenge for future research.

Summary updated

Abstract

We consider the problem of comparing a reference distribution with several other distributions. Given a sample from both the reference and the comparison groups, we aim to identify the comparison groups whose distributions differ from that of the reference group. Viewing this as a multiple testing problem, we introduce a methodology that provides exact, distribution-free control of the false discovery rate. To do so, we introduce the concept of batch conformal p-values and demonstrate that they satisfy positive regression dependence across the groups [Benjamini and Yekutieli, 2001], thereby enabling control of the false discovery rate through the Benjamini-Hochberg procedure. The proof of positive regression dependence introduces a novel technique for the inductive construction of rank vectors with almost sure dominance under exchangeability. We evaluate the performance of the proposed procedure through simulations, where, despite being distribution-free, in some cases they show performance comparable to methods with knowledge of the data-generating normal distribution; and further have more power than direct approaches based on conformal out-of-distribution detection. Further, we illustrate our methods on a Hepatitis C treatment dataset, where they can identify patient groups with large treatment effects; and on the Current Population Survey dataset, where they can identify sub-population with long work hours.
BibTeX
@misc{lee2025finding,
  author = {Yonghoon Lee and Edgar Dobriban and Eric Tchetgen Tchetgen},
  title = {{Finding Distributions that Differ, with False Discovery Rate Control}},
  year = {2025},
  eprint = {2505.13769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.13769},
  howpublished = {arXiv},
}
Figure