Scientists increasingly ask hundreds of questions of the same form at once: is this neuron's activity related to that one once the animal's position is accounted for, or is this genetic marker related to disease once the other markers are accounted for? Each question is a conditional independence test. The test decides whether $X$ and $Y$ are unrelated once a third variable $Z$ is known, and in practice it depends on a machine-learning model of how $X$ or $Y$ behaves given $Z$. When that model is imperfect, the test mistakes its own modeling error for a real relationship. Testing many hypotheses at once then multiplies these false alarms. The two problems have been studied separately. Sequential tests that are robust to model error, such as SKCI, handle one hypothesis at a time. Multiple-testing procedures, such as e-BH, assume each input is already valid.
We propose a different split of the work. The false discovery rate (FDR) is the expected fraction of claimed discoveries that are false. Under e-BH, the FDR depends on the average amount by which the individual tests are miscalibrated, not on the worst one. So the robustness margin that each test currently pays separately can be pooled into one family-level budget and spent where the learned models are weakest. Streams whose conditional models are accurate can then bet almost freely. The proposal has three parts: the allocation rule, the guarantee it comes with, and an experiment that checks whether this beats simply running a robust test per hypothesis and feeding the results to e-BH.
Core Claim
An e-value is a nonnegative test score $E$ whose expectation under the null hypothesis is at most 1. It is typically the final wealth of a gambler who bets against the null. e-BH sorts $K$ e-values and rejects the top $k^*$ hypotheses. Its standard proof bounds each null hypothesis $j$'s contribution to the FDR by $\alpha E_j / K$, where $\alpha$ is the target FDR. That proof gives, under arbitrary dependence between the tests,
$$\mathrm{FDR} \le \frac{\alpha}{K}\sum_{j \in \mathcal{H}_0} \mathbb{E}[E_j] \le \alpha\left(\pi_0 + \frac{1}{K}\sum_{j\in\mathcal{H}_0}\delta_j\right),$$
where $\mathcal{H}_0$ is the set of true nulls, $\pi_0 = |\mathcal{H}_0|/K$ is the fraction of nulls, and $\delta_j = \max(0, \mathbb{E}[E_j]-1)$ is the excess expected wealth that test $j$ picks up because its conditional model is wrong.
Two consequences follow. First, misspecification degrades e-BH linearly and through an average, not exponentially as in the familiar formula $1-(1-\alpha)^K$ for unadjusted tests. Second, a guarantee of $\mathrm{FDR} \le \alpha$ requires only that the family-level average excess stays inside a budget $\eta$, with e-BH run at level $\alpha/(1+\eta)$. Each $\delta_j$ need not be zero.
The new object is a per-stream calibration allocation. Each stream $j$ gets an excess allowance $\eta_j$, chosen from a held-out upper bound on its regression error $\hat\varepsilon_j$, subject to $\frac{1}{K}\sum_j \eta_j \le \eta$. The allowances are set by water-filling: streams with small $\hat\varepsilon_j$ get light correction, and the budget is spent where the corrections cost the least power. The tightness of per-stream calibration is a quantity no prior paper has tuned. Our claim is that loosening it at the family level, and trading that against a slightly lower e-BH level, buys power without losing FDR control.
Research Program
The base test is a sequential kernel test that bets on conditional independence. We will use SKCI's self-normalized kernel payoff with its "truncate-and-shift" calibration. SKCI is a betting test that keeps expected wealth from inflating under the null when the conditional distribution is learned online. It supplies the method we reuse. Its truncation and shift parameters currently aim for zero excess in each stream. We will make them a function of a target allowance $\eta_j$.
The first theoretical step is to derive, or bound numerically, the map from regression error $\varepsilon$ and allowance $\eta_j$ to the required shift. This is the curve that makes water-filling possible.
The second step is a guarantee of the form $\mathrm{FDR} \le \alpha + \Pr(\text{some } \hat\varepsilon_j \text{ underestimates } \varepsilon_j)$. The failure probability is controlled by a held-out estimate of each regression's error, with a union bound or an e-value-based confidence bound across streams.
The third step addresses correlated errors. When hypotheses share coordinates of $Z$, He et al. show that regression errors are correlated and statistic inflation is amplified. The expectation bound above remains valid under such correlation, but the variance of the realized false discovery proportion grows. We will measure that variance directly.
We restrict to local stopping, where each stream stops on its own clock, or to settings meeting the Markov condition of stopped e-BH. Ru et al. show that a global stopping time lets one stream peek into another's future. Combining that leakage with estimation error is out of scope, and we say so.
Prior work places this as follows. Wang and Ramdas supply the aggregation and assume valid inputs. We supply inputs that are deliberately allowed to be slightly invalid, with the invalidity accounted for. SKCI and e-CRT build single-hypothesis betting tests and include no multiplicity correction. CONTRA protects offline p-values against a misspecified $P(X_j \mid X_{-j})$ with a fixed equal-weight mixture across $p$ features. Our allocation is sequential and adaptive across hypotheses rather than fixed. The motivating evidence comes from Roquero Gimenez et al., who show that knockoff FDR control fails entirely under a misspecified Gaussian model, and from He et al., who show that weak regressors push Type-I error from 5% to over 20%. Both show that per-test validity cannot be taken for granted. The impossibility result of Shah and Peters, cited in He et al., constrains us: the guarantee must rest on a structural assumption, here an estimable error bound on each regression, and cannot be distribution-free.
Baselines
The naive reference point is per-stream betting with no robustness correction, combined by e-BH. This matches the DAVT and EC2ST style tests, whose false rejection rates SKCI reports "shoot up toward 100%" on dSprites and Car Insurance even for one hypothesis. It shows how badly FDR fails when model error is ignored.
The strongest competitor is SKCI run independently on each hypothesis, with its default zero-excess calibration, combined by e-BH at level $\alpha$. It is strongest because it is already valid: if each SKCI e-value is valid, e-BH controls FDR under arbitrary dependence, with no further work. On single hypotheses, the SKCI paper reports staying "near or below the target 5% error rate" and is the only method reaching power 1.0 on the 3D separate-coordinate benchmark. We found no reported result on Lacuna for SKCI combined with e-BH across a family of hypotheses, so we will establish that number ourselves. Both arms will use the same regressors, the same streams, the same held-out split, and the same hyperparameter search budget. Beating it means matching its FDR at or below $\alpha$ while making more true discoveries at a fixed sample budget, or reaching the same discoveries with fewer samples.
The offline reference is CONTRA p-values with Benjamini-Hochberg on the variable selection tasks. CONTRA reports 66.67% precision on Celiac markers, against 53.33% for 01-HRT and 57.14% for LL-HRT. A related offline baseline replaces BH with Benjamini-Yekutieli to show the cost of arbitrary-dependence correction.
Two ablations isolate the key ingredient. The first uses a uniform allowance $\eta_j=\eta$ for all streams, with no water-filling. The second uses the budgeted e-BH level without any per-stream loosening. Together they separate the gain from reallocation from the gain from pooling.
Why not just…?
Why not just run SKCI per hypothesis and feed it to e-BH? This is the real threat, and the proposal may lose to it. If the shift-versus-allowance curve is flat near zero excess, so that loosening calibration barely helps power, then lowering e-BH's level to $\alpha/(1+\eta)$ costs more than it buys. The ablation comparing water-filling against $\eta=0$ settles this directly. If per-stream SKCI wins, the contribution reduces to the degradation bound and a measured characterization of family-level error, and we will report it that way.
Why not just use Benjamini-Yekutieli on conditional randomization p-values? BY pays a $\ln K$ penalty that Wang and Ramdas show erodes power. More fundamentally, BY assumes valid p-values, and the evidence from He et al. and CONTRA shows that misspecified conditional models deflate null p-values, which BY cannot repair. The offline baseline arm measures both effects.
Why not just split the sample and fit the conditional models on one half? Splitting removes overfitting leakage but not misspecified model structure. He et al. find that structured residual dependence persists on held-out data from a model class that is too weak, and splitting also halves the testing data. We already use a small split, but only to bound each regression's error, not to make the tests valid. The no-correction arm with honest splitting shows how much error survives splitting.
Proposed Experiments
We will start with synthetic families built from the 3D coordinate benchmark of He et al. and SKCI. Families of $K \in \{10, 100, 1000\}$ hypotheses will share or separate coordinates of $Z$, and regressor quality will be deliberately mixed, using early-stopped and fully trained regressors, so that water-filling has something to exploit. For the real-world-style family, we will use RatInABox, the 100-dimensional neural simulator from the SKCI paper, and test all cell pairs given position. Here the shared conditioning variable produces correlated estimation errors. For variable selection, we will use CONTRA's orng, xor, orng-c, and xor-c settings and the Celiac data, run as streams.
The metrics are realized FDR with its variance across replicates, power, and samples to discovery. The decisive check has two conditions, both in the mixed-quality regime. The proposal must hold FDR at or below $\alpha$ while per-stream SKCI holds FDR but finds measurably fewer true edges. And when the error bounds are deliberately underestimated, FDR must degrade by no more than the derived bound predicts. If the second condition fails, the bound is wrong. If the first fails, the reallocation idea is wrong.
References
- Sequential Kernel-based Conditional Independence Testing via Adaptive Betting
- False Discovery Rate Control with E-values
- On the Hardness of Conditional Independence Testing In Practice
- Anytime-valid FDR control with the stopped e-BH procedure
- Model-X Sequential Testing for Conditional Independence via Testing by Betting
- CONTRA: Contrarian statistics for controlled variable selection
- Knockoffs for the Mass: New Feature Importance Statistics with False Discovery Guarantees.