Spending Little Privacy on Steering: Adaptive Neyman Allocation Under Differential Privacy

When two groups in an experiment have very different noise levels, giving each group half the sample is wasteful. Neyman allocation, the rule that assigns samples in proportion to each group's standard deviation, can cut the sample needed for a given confidence-interval width by about half. Adaptive versions learn the standard deviations as the experiment runs. Those versions need to look at running outcome statistics, and in clinical trials or user-level A/B tests the outcomes are private. Differential privacy (DP) is a guarantee that any released statistic changes very little when one person's data changes, enforced by adding calibrated noise. Under DP, every look at the data costs part of a fixed privacy budget.

This note argues that steering an experiment toward the Neyman ratio should cost very little privacy, because the precision of the final estimate barely depends on getting the allocation exactly right. Most of the budget should go to reporting the treatment effect. A small budget slice, spent early and at doubling intervals, should go to estimating the variance ratio. Separately, wobble-mode observation in gamma-ray astronomy points to a failure mode that adaptive allocation creates under drift, and it suggests a fix.

Core Claim

Write $\pi$ for the fraction of $N$ users assigned to arm A, and $\sigma_A, \sigma_B$ for the outcome standard deviations. The difference-in-means variance is $V(\pi) = \sigma_A^2/(N\pi) + \sigma_B^2/(N(1-\pi))$. It is minimized at the Neyman point $\pi^* = \sigma_A/(\sigma_A+\sigma_B)$, and a second-order expansion around that point gives

$$\frac{V(\pi^*+\delta)}{V(\pi^*)} - 1 \;\approx\; \frac{\delta^2}{\pi^*(1-\pi^*)}.$$

The cost of a misallocation $\delta$ is therefore quadratic, while the benefit of being near $\pi^*$ rather than at $1/2$ is large when the arms are very unequal. For $\pi^* = 0.8$, sitting at $1/2$ inflates variance by roughly 56%, consistent with the roughly 50% sample savings reported by Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing.

A DP-noised variance estimate from $n$ users with steering budget $\epsilon_s$ has privacy error of order $1/(n\epsilon_s)$ on top of sampling error of order $1/\sqrt{n}$. The privacy part stops mattering once $\epsilon_s \gtrsim 1/\sqrt{n}$, and its effect on $V$ enters only squared. The claim is that the optimal steering share of the budget is small and shrinks with sample size. Steering is worth paying for only through a handful of releases, not at every step.

Two further corrections make this work in practice. First, early noisy ratio estimates must be shrunk toward balance by an amount set by their privacy-inflated uncertainty. Second, the variance ratio must be estimated so that both arms are compared over the same time profile. The second correction is where astronomy enters.

Where Each Direction Sits

Neyman allocation is the core mechanism. Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect introduces ClipSMT. ClipSMT plugs running second moments into the Neyman formula and keeps assignment probabilities away from 0 and 1 using the clip $c_t = \tfrac12 t^{-1/3}$, and the paper proves logarithmic regret against the oracle allocation. CLIP-OGD: An Experimental Design for Adaptive Neyman Allocation in Sequential Experiments gets $O(\sqrt{T})$ regret via online gradient descent. The $\mathcal{P}_2$ policy of Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing tracks running standard deviations. All three assume the allocator sees raw outcomes, and that assumption is exactly what we remove.

Dynamic privacy budget allocation supplies the constraint and the scheduling question. Work on DP training, such as Differentially Private Model Publishing for Deep Learning, varies noise over time for a single optimization trajectory. Those schedules decide when to spend; ours decides what to spend on, steering versus reporting. Fully-Adaptive Composition in Differential Privacy supplies the accounting. Its privacy filters are stopping rules that halt before the total budget is exceeded, and they stay valid when per-step budgets are chosen from data. We need that because our stopping time and steering releases are data-dependent.

Offline DP causal inference, including Causal Inference under Differential Privacy: Challenges and Mitigation Strategies and Differentially private learners for heterogeneous treatment effects, studies static noise on fixed datasets. That work motivates the problem, but none of it adapts assignment.

The research file frames one open question as how to split the privacy budget between treatment and control. Under user-level DP that framing is mostly wrong. Users in different arms are disjoint, so the two arms' releases compose in parallel and each can use the full budget. The real split is between steering and reporting queries on the same users, and across repeated running-sum releases that touch the same users many times.

Wobble mode is the methodological lens, not the application. In sequential ON/OFF observing, the telescope points at the source, then at empty sky. Because the two looks happen at different times, atmospheric and zenith drift biases the background estimate (Observation of Gamma Rays from the Galactic Center with the MAGIC Telescope; MAGIC observations of PG 1553+113 during a multiwavelength campaign in July 2006). Wobble mode instead points slightly off the source and records signal and control regions simultaneously (Observations of the magnetars 4U 0142+61 and 1E 2259+586 with the MAGIC telescopes). It also alternates which side is which to cancel fixed camera asymmetries (Mapping the extended TeV source HESS J1857+026 down to Fermi-LAT energies with the MAGIC telescopes).

Adaptive allocation quietly recreates the ON/OFF problem. Once $\pi_t$ moves, arm A's samples cluster at different times than arm B's. Under diurnal or platform drift, full-history per-arm variances then compare different periods, and the estimated ratio is biased even when the true ratio is constant. The wobble-style fix is to estimate the ratio from contemporaneous, propensity-reweighted windows. Inverse-propensity weights give both arms the same time profile, and any drift that scales both arms' noise together cancels in the ratio, just as transparency changes cancel between simultaneous ON and OFF regions. We do not claim anything new for telescope scheduling. Wobble positions already observe both regions simultaneously, so the "50/50 dwell" framing in the file does not straightforwardly apply there.

Research Program

The method, which we call PrivNeyman, runs in doubling epochs. At the end of each epoch it releases, under zero-concentrated DP (a variant suited to composing many Gaussian releases), noisy per-arm first and second moments for that epoch's users only. Each user then appears in exactly one steering release. The releases use outcomes bounded in $[0,B]$ and reweight by inverse propensity. The allocator computes a private Neyman ratio together with a confidence interval for it. That interval is widened by the known Gaussian noise variance, adapting the time-uniform variance bounds of Sharp Empirical Bernstein Inequalities for the Variance of Bounded Random Variables.

The next epoch's assignment probability is the endpoint of that interval closest to $1/2$. This replaces ClipSMT's fixed clip with a clip set by the data's uncertainty, which grows when privacy noise is large. Assignments are post-processing of the steering releases and cost no further privacy. Reporting releases noisy running means through a standard tree-based continual-release mechanism, and a privacy filter in the style of Fully-Adaptive Composition in Differential Privacy governs data-dependent stopping when a target interval width is reached.

The analysis has three parts. The first derives the excess variance of PrivNeyman relative to the oracle, which knows $\sigma_A, \sigma_B$ and spends the whole budget on reporting. From that bound we read off the optimal steering share $\rho_s/\rho$ as a function of $n$, $\rho$, and $\pi^*$. The second asks whether the regret rate degrades from logarithmic to something that depends on the noise term $1/(n\epsilon_s)$. The third characterizes the drift bias of full-history ratio estimation under adaptive assignment.

We expect one negative finding in the high-privacy regime, and we will report it. In central DP at small $n\epsilon^2$, the reporting noise on each arm mean is set by the bound $B$ rather than by $\sigma_k$, so it is the same size in both arms. Noise that is equal across arms pulls the optimal allocation back toward balance, so adaptation buys little there. The theory should predict the crossover.

Baselines

The naive reference is a balanced 50/50 design that spends the entire budget on reporting the two arm means, with a static per-release budget. It pays nothing for steering and loses the full Neyman gain. Beating it means needing fewer users to reach the same private interval width whenever $\pi^*$ is far from $1/2$. We also need to show where it ties, which is the predicted high-privacy regime.

The strongest existing method is non-private adaptive Neyman allocation, run with the same users and the same stopping rule. For $\mathcal{P}_2$, Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing reports 2,672 samples versus 4,034 for Clip-OGD on a Bernoulli instance, and about a 50% reduction versus balanced randomization on Exponential outcomes with means 10 and 0.1. For ClipSMT we found no reported scalar result on Lacuna, beyond its rate and its qualitative dominance over ClipOGD, explore-then-commit, and balanced allocation. Non-private $\mathcal{P}_2$ and ClipSMT are upper references, not competitors. Success means PrivNeyman's sample count approaches theirs as $\epsilon$ grows: within a few percent at moderate privacy on these exact instances, measured against the private oracle.

The closest private competitor is the obvious construction, DP-ClipSMT. It privatizes the running moments at every step with a uniform per-step budget, uses the original clip, and is accounted with the same filter. It isolates our two ingredients, the doubling-epoch steering schedule and the uncertainty-based clip. Beating it at matched total budget is the central test.

Ablations swap each ingredient back individually: per-step steering, a fixed clip, a fixed 50% steering share, and full-history rather than contemporaneous ratio estimation under drift. A private oracle Neyman design caps achievable performance. All methods share the outcome stream, the random seeds, the total budget, the accountant, and a fixed grid for any tuned constants.

Why not just…?

Why not just run 50/50 and spend all privacy on the answer? When the arms are similar, or privacy is very tight, this baseline may be right, and the experiments are designed to locate that regime rather than hide it. Away from it, the quadratic-loss argument and the roughly 50% non-private gap in Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing imply that a small steering share recovers most of the gain. We sweep $\epsilon$ and $\pi^*$ to map where each design wins.

Why not just do private explore-then-commit with one variance release? Under privacy this is the real threat, because each extra steering release costs budget and the loss is flat near $\pi^*$. Non-privately, Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect found explore-then-commit degrades as $\pi^*$ moves away from 0.5, because a short pilot misjudges extreme ratios. We treat the number of steering releases $K$ as a dial running from 1 (explore-then-commit) through doubling epochs ($\log T$ releases) to every step. If $K=1$ matches doubling epochs, we will recommend it.

Why not just have each user randomize their own response (local DP) and run ClipSMT on the noisy responses? This is legitimate post-processing and needs no new accounting. It also has a clean optimum: with per-response noise variance $c/\epsilon^2$, Neyman allocation on the privatized data becomes

$$n_k \propto \sqrt{\sigma_k^2 + c/\epsilon^2},$$

and ClipSMT tracks it automatically. The price is noise of order $1/(\sqrt{n}\,\epsilon)$ instead of $1/(n\epsilon)$, and an allocation flattened toward balance. We include it as a baseline and expect it to lose badly to central-DP designs except at very small $n$.

Why not just estimate the ratio from public or pilot data? This is the approach of Differentially Private Learning with Small Public Data, which reports $\cadata$ MSE of 0.1027 versus 0.1253 for the next private baseline at $\epsilon=0.3$, steering private training with tiny public sets. Its authors leave distribution shift between public and private data open. We test this approach with public samples drawn from a shifted subpopulation and measure the resulting misallocation $\delta$.

Proposed Experiments

The first testbed is synthetic. It uses the nine ClipSMT instances, plus the Exponential (means 10 vs 0.1) and Bernoulli instances from the $\mathcal{P}_2$ paper, with outcomes bounded or truncated for DP. Total zCDP budgets correspond to $\epsilon$ from 0.1 to 8, and horizons run from $10^3$ to $10^6$.

The second testbed is semi-synthetic, built on CRITEO-UPLIFTv2 from A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling. Its 13.9 million users and validated random assignment let us resample real per-arm outcomes sequentially. Visit and conversion outcomes give naturally heteroscedastic binary arms, and the paper's finding that model rankings separate only above 1M samples tells us which horizons are informative. On top of this we add drift: a multiplicative diurnal factor shared by both arms, then an arm-specific one, to test the wobble-style ratio estimator. The TripAdvisor noncompliance benchmark of Efficient Adaptive Experimentation with Noncompliance is a stretch extension, since compliance adds a second variance source.

The metrics are users needed to reach a target private interval width, ATE MSE relative to the private oracle, and coverage of the anytime-valid intervals. Privacy is verified both through the accountant and with a simple empirical audit that injects canary users and tests whether they can be detected.

Three results would settle the core claim. The first is that the empirically optimal steering share falls with $n$ as the theory predicts and stays small at moderate $n$. If instead it stays near half, the flatness argument fails in practice. The second is that, at matched budget, doubling-epoch steering beats per-step DP-ClipSMT, and either beats or ties explore-then-commit when $\pi^*$ is far from 0.5. The third is that, under shared drift with adaptive assignment, full-history ratio estimation measurably misallocates and inflates variance, while the contemporaneous, propensity-reweighted estimator does not. If that third result fails to appear, we drop the wobble component and report the null.