TL;DR
This paper introduces a new statistical framework for identifying which groups in a dataset differ from a reference distribution while strictly controlling the False Discovery Rate (FDR). By inventing "batch conformal p-values," the authors provide a distribution-free method that allows researchers to reuse reference data across multiple comparisons without compromising statistical validity.
What Problem Does This Paper Address?
In fields like medicine or economics, researchers often need to compare a "control" group (the reference) against many different "treatment" or "sub-population" groups. The goal is to pinpoint exactly which groups deviate from the norm.
This presents two major challenges:
- The Multiple Testing Problem: If you run 100 tests, you're bound to find "significant" differences just by chance. Controlling the False Discovery Rate (FDR)—the expected proportion of false discoveries among all discoveries—is essential.
- Data Scarcity and Dependency: Standard methods often require splitting the reference data into $K$ separate pieces to keep tests independent. If your reference group is small, this destroys your statistical power. However, if you reuse the same reference data for every comparison, the tests become dependent, which usually breaks the math behind standard FDR-control tools like the Benjamini-Hochberg (BH) procedure.
Methods
The authors bridge this gap using Batch Conformal Inference.
- Batch Conformal P-values: Traditional conformal prediction generates a p-value for a single new data point to see if it's an outlier. The authors extend this to "batches" (entire groups). They calculate a p-value based on how a specific quantile (e.g., the median or the 90th percentile) of a comparison group ranks against the reference group.
- Score Functions: The method is flexible; you can use simple values (like blood pressure) or complex "nonconformity scores" (like residuals from a machine learning model) to define what "different" looks like.
- Theoretical Breakthrough: The core contribution is a mathematical proof showing that these batch p-values satisfy a property called Positive Regression Dependence (PRDS). This is a specific type of "good" dependency. Because they proved PRDS holds even when reusing reference data, they can safely apply the standard Benjamini-Hochberg procedure to guarantee exact FDR control.
- Inductive Rank Construction: To prove the PRDS property, the authors developed a novel inductive technique for constructing rank vectors, which may have broader applications in probability theory.
Datasets
The paper validates the method using three distinct approaches:
- Synthetic Simulations: Testing 1D and multivariate shifts under both normal and "heavy-tailed" (Cauchy) distributions to compare power against "oracle" tests (tests that know the true underlying distribution).
- HALT-C Study: A medical dataset tracking Hepatitis C treatment effects. The authors used it to identify specific patient demographics (categorized by age, BMI, and sex) that showed significantly different treatment responses in platelet counts.
- Current Population Survey (CPS): A massive US Census dataset. The authors identified specific sub-populations (defined by education, race, and age) that work significantly longer hours than a reference group.
Key Insights and Ideas
- Reusing Data Safely: The biggest "aha moment" is that you don't have to sacrifice power by splitting your reference set. You can "double-dip" into your reference data for multiple tests as long as you use the batch conformal framework.
- Targeting the Tails: Unlike the popular Wilcoxon rank-sum test, which looks at the "bulk" of the distribution, this method allows researchers to target specific quantiles. For example, you can specifically look for treatments that help the sickest 10% of patients, even if the average patient stays the same.
- Distribution-Free Power: In simulations, this method performed nearly as well as "oracle" tests that had perfect knowledge of the data's distribution, proving that you don't need to assume "normality" to get high-quality results.
Key Results
- Guaranteed FDR Control: The method consistently controlled the FDR at the target level (e.g., 5% or 10%) across all simulations, regardless of how many groups were tested.
- Superior Power: It significantly outperformed "subsampling" approaches (which only use one data point per group) and was more robust than t-tests when the data was non-normal.
- Computational Efficiency: Unlike permutation tests, which can be computationally "expensive" because they require thousands of reshuffles, the batch conformal p-value is calculated using a closed-form combinatorial formula, making it much faster for large-scale data.
Limitations and Future Work
- Hyperparameter Selection: The user must choose which quantile ($\eta_k$) to test. While the median is a safe default, the "best" quantile depends on the specific scientific question, and the paper suggests data-driven ways to tune this.
- Independent Comparison Groups: The current proof assumes that while groups share a reference, they are independent of each other. Future work could explore settings where comparison groups also share data or have internal dependencies.
- Pairwise Comparisons: The current framework is "one-vs-many." Extending this to "many-vs-many" (comparing every group to every other group) remains an open challenge for future research.
Summary updated