In modern data science, we rarely test a single hypothesis. Whether scanning the genome for disease markers or running thousands of A/B tests on a website, we face the "multiple comparisons problem": the more questions you ask, the more likely you are to get a "significant" answer by sheer luck.
The standard solution is to control the False Discovery Rate (FDR)—the expected proportion of false positives among all discoveries. However, traditional methods like the Benjamini-Hochberg (BH) procedure rely on a fragile assumption: that the tests are independent or follow very specific correlation patterns. When tests are "arbitrarily dependent"—meaning they could be correlated in any unknown, complex way—traditional p-value methods become either invalid or overly conservative.
A new framework, pioneered by researchers like Aaditya Ramdas, Ruodu Wang, and Vladimir Vovk, replaces p-values with e-values. This shift allows for rigorous error control under any dependency structure without the heavy "mathematical penalties" that usually cripple discovery power.
The Problem with p-values and Dependence
To understand why e-values are a breakthrough, we must look at how p-values handle dependence. A p-value is a probability-based measure: under the null hypothesis, the probability that a p-value is less than $\alpha$ is at most $\alpha$.
When you have many p-values that are correlated, their joint distribution becomes a nightmare to manage. To guarantee FDR control under "arbitrary dependence," the classic Benjamini-Yekutieli procedure applies a "logarithmic penalty" (a reshaping function) that shrinks the discovery threshold by a factor of roughly $\sum_{i=1}^m (1/i) \approx \ln(m)$. As Aaditya Ramdas and colleagues note in A unified treatment of multiple testing with prior knowledge using the p-filter, these reshaping functions are necessary safeguards but significantly reduce the power to find real signals.
e-Values: Evidence as a Bet
An e-value is an expectation-based measure of evidence. Formally, a random variable $E$ is an e-value if its expected value under the null hypothesis is at most 1 ($E[E] \le 1$).
The intuition, as explained in Game-theoretic statistics and safe anytime-valid inference by Glenn Shafer, Peter Grünwald, and Vladimir Vovk, is to think of testing as a game. You start with \$1 and bet on an outcome. If the null hypothesis is true, you don't expect to make money. If your "betting score" (the e-value) ends up being 100, you have strong evidence against the null, because it is unlikely you would have multiplied your wealth a hundredfold by chance.
The mathematical magic of e-values lies in the linearity of expectation. The average of several e-values is itself an e-value, regardless of how they are correlated. This property makes them naturally robust to dependence.
The e-BH Procedure
In False discovery rate control with e-values, Aaditya Ramdas and Ruodu Wang introduced the e-BH procedure. It mirrors the classic BH procedure but operates on e-values:
- Rank your e-values from largest to smallest: $e_{(1)} \ge e_{(2)} \ge \dots \ge e_{(m)}$.
- Find the largest $k$ such that $e_{(k)} \ge \frac{m}{k \alpha}$.
- Reject the top $k$ hypotheses.
The breakthrough is that this simple procedure controls the FDR under arbitrary dependence without the logarithmic penalty required by p-values. Because e-values are "expectation-constrained" rather than "probability-constrained," they don't need to be "reshaped" to account for unknown correlations. This allows researchers to maintain higher discovery power in complex datasets where correlations are unknown or difficult to model, as discussed in Hypothesis testing with e-values.
Online and Adaptive Testing
The robustness of e-values is particularly useful in online multiple testing, where hypotheses arrive in a stream and decisions must be made in real-time.
In Online multiple testing with e-values, Ziyu Xu and Aaditya Ramdas show that e-value-based algorithms (like e-LOND) eliminate the need for conservative decaying thresholds. In traditional online p-value testing, the "discovery budget" must be spread very thinly to account for potential future dependencies. With e-values, the martingale properties of the evidence allow for more aggressive testing.
This framework has been extended to even more complex scenarios:
- Bandit Multiple Testing: Ziyu Xu and colleagues in A unified framework for bandit multiple testing use e-processes to decouple data exploration from discovery, allowing for adaptive stopping rules.
- Accept-to-Reject (ARC): Lasse Fischer and others in An online generalization of the (e-)Benjamini-Hochberg procedure developed methods that allow researchers to change their minds—rejecting a previously accepted hypothesis as more evidence accumulates—while still maintaining rigorous error control.
Why It Works: Intuition over Formulas
The core reason e-values work where p-values fail is their relationship to the arithmetic mean.
If you have $m$ null hypotheses, and you use p-values, the probability that any of them are small depends heavily on how they overlap. If they are perfectly correlated, the chance of one being small is the same as the chance of all being small. If they are independent, the chances add up. To be safe, p-value methods must assume the worst-case overlap.
E-values, however, are bounded by their average. If the average of $m$ e-values is $E_{avg}$, then by Markov’s inequality, the probability that $E_{avg} > 1/\alpha$ is at most $\alpha$. This holds no matter how the individual e-values are related. By framing evidence as a quantity that can be averaged rather than a probability that must be bounded, e-values provide a "safe" currency for statistical discovery in the messy, interconnected reality of modern data.