Every Sample Tests Once, Then Trains: Self-Tuning Sequential Two-Sample Tests

When a sensor array, a clinical cohort, or an image pipeline starts producing data that looks "off," engineers want three answers: is the new data really different from the old, by how much, and in which features. A popular statistical tool for the first question trains a classifier to tell old samples from new ones and declares a shift if it does better than chance. This approach pays two hidden costs. First, the data must be split: part trains the classifier and the rest tests it, so a large fraction of scarce samples never contributes to the final verdict. Second, the classifier's hyperparameters (regularization strength, feature weights) are tuned in an expensive outer loop that repeatedly retrains models, on yet another held-out portion. In the small-sample regimes where fault diagnosis matters most, these two costs compound.

We propose a test in which every sample plays three roles in a fixed order, so nothing is thrown away. Data arrive in batches. Each new batch is first used to test the current classifier, which was trained only on earlier batches. It is then used as validation data for a single gradient step on the classifier's hyperparameters. Only after that is it added to the training data. Because the test statistic is a running product of "bets" that each depend only on the past, the test stays valid however the hyperparameters adapt. Its growth rate also estimates a divergence between the two distributions. Per-feature relevance hyperparameters, tuned along the way, show which features drive the shift. The result is one training run that detects the shift, sizes it, and localizes it, without a data split and without an outer tuning loop.

Core Claim

The key observation is that sequential testing with e-values already avoids discarding data, and it also supplies a clean validation set for hyperparameter tuning at no extra cost. An e-value is a nonnegative statistic whose expectation under "no shift" is at most one, so products of them over time stay valid at any stopping point. E-Valuating Classifier Two-Sample Tests built the sequential protocol: each batch is scored by a classifier trained on preceding data. It reports that small batches cause unstable classifiers and weak e-values, and that retraining across batches is costly. Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters tunes continuous regularizers inside a single training run using gradients computed on a held-out validation set $T_2$, at roughly 30% overhead. That method still requires holding $T_2$ out. In a sequential test, the incoming batch is exactly such a held-out set, at the moment before it joins training. Merging the two removes the validation split. The tuning also targets the specific failure E-C2ST reports: a regularization level fixed for the final sample size is too weak in early, tiny batches. A hyperparameter tuned online can start strong and relax as evidence accumulates.

Concretely, we pair each new sample $x$ from the reference distribution $P$ with a sample $x'$ from the new distribution $Q$, and let $p_\theta(x)$ be the classifier's probability that $x$ came from $Q$. The bet on a pair is

$$e = \frac{2\,(1-p_\theta(x))\,p_\theta(x')}{(1-p_\theta(x))\,p_\theta(x') + p_\theta(x)\,(1-p_\theta(x'))},$$

which is twice the classifier's probability of ordering the pair correctly. If $P = Q$, the ordering is a fair coin given the pair, so $E[e]=1$ for any $\theta$ fixed by past data. The wealth $W_t$, the product of all bets so far, can therefore be monitored continuously, and rejecting when $W_t \ge 1/\alpha$ controls Type-I error at level $\alpha$ by Ville's inequality. The weights $\theta$ are trained on the past with regularization hyperparameters $\eta$. After a batch is scored, $\eta$ takes one hypergradient step that increases that batch's log-wealth through the one-step weight update, using the greedy, identity-Hessian approximation of the $T_1$–$T_2$ method. Only then does the batch enter training.

Two byproducts follow. The average log-bet $\frac{1}{t}\log W_t$ is an estimate, from below, of a mutual-information-type divergence between $P$ and $Q$; it equals the Jensen–Shannon divergence when the classifier is Bayes-optimal and stays below it otherwise. A time-uniform confidence interval on this rate therefore gives a shift magnitude with error bars, which descriptive dataset distances do not provide. Second, if $\eta$ includes per-feature relevance scales, such as automatic-relevance-determination lengthscales or per-input weight decay, the tuned scales act as a running map of which features the detector relies on.

This differs from the closest work. AutoML Two-Sample Test connects all four directions, but it uses a two-stage split and brute-force outer-loop AutoML. Our method has no split and no outer loop, though it covers only continuous hyperparameters, not model-family search.

Research Program

The first component is the test itself, for two witness families. The first is a kernel logistic or kernel Fisher-type witness with relevance-weighted inputs, close to the precision-weighted witness of A Witness Two-Sample Test. That paper uses a 50/50 split and heuristic choices of $\lambda$ and bandwidth; here those are online hyperparameters. The second is a small neural network with per-input gates and weight decay, warm-started across batches rather than retrained from scratch. We will prove that the wealth process is a nonnegative supermartingale under the null for any hyperparameter rule that is predictable, meaning it depends only on past batches. This is straightforward, but it must be stated carefully, because the hypergradient step uses the current batch only after that batch has been bet on. We will also give a regret-style bound relating the growth rate to the divergence for well-specified witnesses.

The second component is an analysis of the early-batch regime. We will measure how the tuned regularization path evolves with $t$. We will test whether online tuning removes the weak-e-value problem E-C2ST reports at batch sizes near 8, by comparing against the same procedure with $\eta$ frozen at values chosen for the final $n$.

The third component evaluates attribution. Relevance scales are descriptive and come with no false discovery guarantee. We will therefore evaluate them as localization scores against known ground truth. We will also test a cheap validity layer: after the global test rejects, the learned relevance ordering is fixed using past data only, and a few top features are tested sequentially by betting with a witness that excludes each feature. This yields e-values per feature that can be combined with an e-value-based false discovery procedure. Whether this layer has usable power is an open empirical question, and we treat it as one.

Baselines

The naive reference is a classifier two-sample test with a 50/50 split and default hyperparameters, as in Revisiting Classifier Two-Sample Tests. It is run on the same total sample budget with the same model class. Beating it means higher power at equal $n$.

The direct parent is E-C2ST from E-Valuating Classifier Two-Sample Tests, with hyperparameters fixed by a reasonable grid chosen on a separate pilot set. It shares our validity mechanism, so it isolates what online tuning adds. On Blobs it is reported to reach 100% rejection faster than the sequential baselines S-C2ST and M-C2ST while keeping Type-I error below 0.05. Beating it means a shorter expected stopping time at the same Type-I level and higher power at small batch sizes, using the same data stream and equal or lower wall-clock compute.

The strongest split-free fixed-sample competitor is MMD-FUSE from MMD-Fuse: Learning and Combining Kernels for Two-Sample Testing Without Data Splitting. It learns kernels on pooled data in a way that ignores which sample came from where, then fuses many kernel statistics. On CIFAR-10 versus CIFAR-10.1 it reports power 0.937, against 0.883 for MMDAgg (MMD Aggregated Two-Sample Test) and 0.744 for the split deep-kernel test MMD-D. We will run it at the identical total sample size, and run our test in a single pass over the same samples. Matching 0.937 within noise would show that the sequential, attributing test loses nothing on global detection. Beating it is not required for the claim, but losing badly would narrow the claim to settings that need anytime monitoring or localization.

For the tuning-cost axis, the competitor is the AutoML Two-Sample Test (AutoML Two-Sample Test), run with the same time limit as our full single pass, including its one-minute setting. A split classifier tuned with BOHB (BOHB: Robust and Efficient Hyperparameter Optimization at Scale) serves as an outer-loop reference. No reported power numbers for these on our benchmarks were found on Lacuna, so we will run them ourselves.

For attribution, the competitor is Knockoff-C2ST: False Discovery Controlled Attribution of Distribution Shift. On UCI Heart Disease it keeps FDR below 1% at a nominal 10%, but its power is often near zero without stability aggregation, while marginal tests reach FDR as high as 95%.

The key ablation removes only the hypergradient step and keeps everything else identical. A second ablation feeds the hypergradient from a separately held-out validation slice, to show that the free validation role, not tuning per se, carries the gain.

Why not just…?

Why not just use MMD-FUSE, which already avoids splitting and has the best reported CIFAR-10.1 power? This alternative is a real threat on fixed-sample global detection, and we may not beat 0.937 there. It falls short on two other requirements. It is a fixed-sample test, so monitoring a live stream would need repeated testing with corrections. Its kernels are learned without using sample labels, so no supervised per-feature relevance comes out of it. The CIFAR-10.1 comparison at equal $n$ settles how much global power the sequential design costs. The stopping-time and localization experiments test whether that cost buys something MMD-FUSE cannot provide.

Why not just run E-C2ST with well-chosen fixed hyperparameters, or simply warm-start it? If a good fixed $\lambda$ works as well, our tuning mechanism is unnecessary, and the frozen-$\eta$ ablation is built to show exactly this. The motivating evidence is E-C2ST's own report of weak e-values at small batch sizes. A single fixed regularizer cannot suit both 16 and 2,000 training samples. If frozen hyperparameters with warm-starting match our stopping times across Blobs, HIGGS and MNIST, we will report that and recast the contribution as the divergence estimate and the attribution layer.

Why not just cross-fit a classifier test, rotating which fold is held out, so all data are used? Cross-fitted classifier statistics need either permutation calibration, which retrains every fold for every permutation, or asymptotic variance arguments that break down near the null. Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference addresses that breakdown with an overlap parameter $\tau$ that must be set by hand. We will include a K-fold cross-fitted C2ST with permutation calibration at matched compute. Our expectation is that it is competitive in power but several times more expensive and not valid under continuous monitoring.

Why not just test each feature separately for attribution? Knockoff-C2ST shows that marginal tests reach up to 95% false discoveries under correlated features. On Blobs, marginal means and variances are identical, so per-feature tests have no power at all.

Proposed Experiments

The decisive validity check runs every method under the null, using permuted CIFAR-10, HIGGS background against itself, and Blobs with identical parameters. We require Type-I error at or below 0.05 under continuous monitoring with adaptive hyperparameters, over at least 1,000 repetitions.

The decisive efficiency check uses Blobs, HIGGS, and corrupted or downsampled MNIST. It measures expected stopping time and power at fixed sample budgets for batch sizes 8, 32 and 128, against E-C2ST and its frozen-$\eta$ ablation. It also measures wall-clock compute against AutoTST and BOHB-tuned split tests. CIFAR-10 versus CIFAR-10.1 provides the head-to-head with MMD-FUSE's reported 0.937.

For shift magnitude, we use synthetic Gaussian shifts where the Jensen–Shannon divergence can be computed numerically. There we check whether the confidence interval on the growth rate covers a valid lower bound and tightens as expected. On the CIFAR-10 augmentation shifts used by Lightspeed Geometric Dataset Distance via Sliced Optimal Transport, we report Spearman correlation between our rate and s-OTDD. That paper reports a Spearman correlation with exact OTDD of about 0.93–0.99, in 53 seconds compared with over 20 hours for exact OTDD. Our rate measures discriminability rather than transport geometry, so agreement is informative but not the target.

For localization, we perturb known features in UCI Heart Disease (cholesterol, blood pressure, resting heart rate) and known pixel regions in MNIST. We score the area under the ROC curve of the tuned relevance scales against ground truth, and compare power and FDR of the sequential per-feature layer against Knockoff-C2ST. The proposal succeeds if three things hold: validity holds under monitoring, online tuning beats frozen hyperparameters in the small-batch regime at no greater compute, and relevance scales localize shifts better than marginal tests while the per-feature layer shows nonzero power where Knockoff-C2ST reports near zero.