Randomized experiments are analyzed with messy tables: baseline covariates arrive with missing values, typos, unit errors and swapped fields. An analyst would like to clean those covariates aggressively, choosing among many cleaning pipelines by checking which one best predicts the outcome. Doing so should make treatment-effect tests more powerful. The usual worry is that it amounts to data snooping and voids the experiment's guarantees.
This note argues that the worry has a precise and favorable answer. Consider a Fisher randomization test, which judges significance by recomputing a statistic under re-drawn treatment assignments from the known design. Under the null hypothesis of no effect for any unit, the outcomes do not depend on who was treated. So any cleaning procedure that may look at covariates and outcomes, but never at treatment labels, produces a cleaned table that stays fixed across re-randomizations, and the test remains exact. We propose to build that procedure, which we call treatment-blind, outcome-aware pipeline search. We will prove its validity, extend it to confidence intervals, and test whether it buys real power over cleaning that ignores outcomes.
Core Claim
Fix the potential outcomes, the fixed responses each unit would show under treatment or control. Fix the raw dirty covariate matrix $X$, and let $W$ be the random treatment vector drawn from the known design $P(W)$. Let $\hat\pi$ be a cleaning pipeline chosen by any search that sees $(X, Y)$ but not $W$. Examples include picking imputation, outlier repair and missingness indicators to minimize cross-validated error predicting $Y$ from cleaned covariates. Let $T$ be any statistic, such as a Lasso-adjusted difference in means, computed on the cleaned table $\tilde X = \hat\pi(X, Y)$. Under the sharp null $Y_i(1) = Y_i(0)$ for all $i$, the observed $Y$ is identical for every possible assignment. That makes $\tilde X$ a constant, and the randomization p-value
$$ p = P_{W' \sim P(W)}\big( T(W', Y, \tilde X) \ge T(W, Y, \tilde X) \big) $$
satisfies $P(p \le \alpha) \le \alpha$ exactly, for any search however adaptive.
For a constant-effect null $Y_i(1) - Y_i(0) = \tau$, the control outcomes $Y - \tau W$ are known and fixed. Rerunning the blind search on $(X, Y - \tau W)$ and inverting over a grid of $\tau$ values yields exact confidence sets.
The empirical claim is the substantive one. Which cleaning step helps is strongly dataset-dependent, a central finding of CleanML. We therefore expect outcome-aware selection among pipelines to give materially shorter intervals than any single fixed cleaner. We also expect it to give materially shorter intervals than cleaning chosen from covariates alone.
Position Relative to Prior Work
Two papers motivate the gap. Conformal Data Cleaning detects and repairs corrupted cells by predicting each column from the others with calibrated conformal intervals. It reports gains only on prediction metrics and says nothing about treatment-effect validity. Run naively on a full experimental table that includes the treatment column, it imputes using $W$, which is exactly what our analysis forbids. Learn2Clean+HIL searches over sequences of cleaning actions with reinforcement learning, rewarding downstream accuracy. We reuse that kind of search space but change both the reward and the information the search may see.
Three papers supply the method. Lasso adjustment in block and rerandomized experiments gives the adjusted estimator we use as the test statistic, but it assumes clean, fully observed numeric covariates. Conformal Selective Borrowing is the closest structural precedent. It lets an adaptive, possibly wrong data-selection step (filtering external control rows) feed a Fisher test while keeping exact Type I error. It filters whole rows from an external source, whereas we repair cells inside the trial itself, and it assumes those cells are already clean. The AutoML two-sample test shows that minimizing squared prediction error of a witness function maximizes a permutation test's signal-to-noise ratio. We borrow that logic to justify outcome-prediction error as the search objective, since residual variance governs the power of an adjusted randomization test.
One prior result constrains us. The design-based asymptotics of the Lasso paper treat covariates as fixed. Under the weak null of zero average effect, the outcomes do vary with $W$, so outcome-aware cleaning makes $\tilde X$ random, and the paper's Neyman-style intervals are not automatically valid. We do not claim otherwise. For average-effect inference we use cross-fitting: the pipeline is chosen on one fold's $(X, Y)$ and applied to the other. Calibrating this by simulation is a separate, weaker result.
Research Program
The first deliverable is the validity result: exactness of blind cleaning for sharp and constant-effect nulls under complete, blocked and rerandomized designs. It comes with a clean counterexample showing that any cleaner touching $W$ can inflate Type I error. One example is imputing a covariate from a model that includes the treatment indicator; another is selecting the pipeline that maximizes the observed effect.
The second deliverable is the search itself. Its space contains mean/median imputation with missingness indicators, conformal cell repair with column models trained without $W$, winsorization, and categorical normalization. It is scored by cross-validated prediction of $Y$ (or of $Y - \tau W$) from cleaned covariates. Test inversion needs one search per $\tau$ grid point. To control cost, we warm-start each search from its neighbor's choice and record how often the selected pipeline changes across $\tau$.
The third deliverable is a semi-synthetic benchmark of dirty randomized experiments. None exists in the material we surveyed, so we construct one. We take the 16 OpenML tables used by Conformal Data Cleaning and the 14 CleanML tables with real errors. Treatment is assigned by simulated complete and blocked randomization. Outcomes are built from the clean version of each table plus a known effect, and the analysis sees only the dirty version. Corruption comes from Jenga (swaps, scaling, Gaussian noise, categorical shifts) at 1–40%, plus CleanML's natural errors. As a real-data anchor, we inject corruption into the covariates of the Opportunity Knocks block experiment analyzed in the Lasso paper.
Baselines
The naive reference point is the unadjusted difference in means tested by Fisher randomization. It ignores covariates entirely, is trivially valid, and marks the floor for power.
The strongest existing practical method is the textbook recipe of mean imputation plus missingness indicators, followed by Lasso adjustment (Zhu, Liu and Yang). CleanML found that sophisticated cleaners like HoloClean showed no noticeable advantage over mean/median imputation, so this recipe is what a careful applied statistician would actually run. The only reported number we found for it is on clean covariates: the Lasso paper reports 54–77% reductions in standard deviation and root mean squared error in simulations, and an 18% shorter confidence interval in one application. There is no reported result on Lacuna for any design-based method on corrupted covariates.
The second strong competitor is conformal cell repair applied to covariates only, without $W$ or $Y$. It is valid because $X$ is fixed. Its source paper reports a downstream improvement in 61% of cases and over 15% improvement at 30–40% corruption, both on prediction tasks rather than effect estimation.
Every method shares the same dirty table, the same final Lasso-adjusted statistic, the same 2,000 permutation draws, and the same wall-clock budget for any search. Every method also gets the same candidate pipeline pool, so outcome-aware search wins only by choosing better among options the baselines could also have used.
Two ablations isolate the key ingredient. The first, covariate-only selection, runs the same search but scores pipelines by covariate reconstruction error instead of outcome prediction. It tests whether looking at $Y$ is what helps. The second, treatment-aware search, lets the search use $W$. Run once on observed data it shows the Type I error inflation; rerun inside every permutation draw it is valid but roughly 2,000 times more expensive. This tells us whether blindness costs any power relative to the full-recomputation approach.
The oracle analyzes the clean table. Beating the strongest baseline means two things together. Type I error must stay within Monte Carlo error of 0.05 in every cell. Median interval length must be at least 10% shorter at matched coverage on a majority of tables at corruption levels of 10% and above, with a paired randomization test across tables.
Why not just…?
Why not just mean-impute, add missingness indicators, and let Lasso adjustment absorb the rest? This is the most serious threat. CleanML's evidence that simple imputation rarely loses on prediction suggests it may suffice here too. The counter-evidence is also from the file: at 30–40% corruption, Conformal Data Cleaning reports that uncalibrated baselines degrade performance while calibrated repair holds over 15% gains. Measurement error attenuates the adjustment coefficients that drive variance reduction, which points to a gap at high corruption. The benchmark measures that gap directly across corruption levels. If it is negligible below 20% corruption, we will report the method as useful only in heavily corrupted regimes.
Why not just clean covariates with a good cleaner that never sees outcomes, since that is valid anyway? It is valid, and it is our covariate-only ablation. The reason to expect it to fall short is that reconstruction-optimal cleaning need not be outcome-relevant. A repair that fixes a column with no bearing on $Y$ costs nothing but helps nothing. A winsorization that destroys an informative extreme value can hurt, echoing CleanML's finding that outlier cleaning and deduplication are often neutral or harmful. If the ablation matches the full method, the honest conclusion is that the value lies in the validity analysis, not the outcome-aware search.
Why not just rerun any cleaning search inside every permutation, which is valid without any blindness argument? This "analyze as you randomize" route is correct and may even be more powerful, because it can target the effect statistic directly. Its cost is one full pipeline search per permutation draw, and per $\tau$ for intervals; the file already lists computational scalability as a limitation of exact randomization methods. The treatment-aware-inside-permutation ablation quantifies both its power edge and its cost. If the power edge is large, we will treat blind search as a cheap approximation rather than a replacement.
Proposed Experiments
The decisive checks come in three tiers. First, a validity check: across 1,000 re-randomizations per table under the sharp null, blind search and both valid baselines hold rejection at 0.05, while single-pass treatment-aware cleaning visibly exceeds it. If blind search ever exceeds it beyond Monte Carlo error, the core claim fails. Second, a power check: plot power at a fixed effect and median inverted-interval length against corruption rate for every method and the clean-table oracle, on OpenML/Jenga and CleanML tables. We also compute the fraction of the oracle's improvement over unadjusted analysis that each method recovers. Third, a real-data check on the corrupted Opportunity Knocks covariates, using its block design. We report interval length relative to the Lasso analysis of the clean data, and coverage of the clean-data point estimate across 200 independent corruption draws.
A secondary analysis tests the weak-null cross-fitted variant for coverage of the true average effect. It is reported separately because its guarantee is only asymptotic and simulation-calibrated.
References
- CleanML: A Benchmark for Joint Data Cleaning and Machine Learning [Experiments and Analysis]
- From Data Imputation to Data Cleaning - Automated Cleaning of Tabular Data Improves Downstream Predictive Performance
- Active Reinforcement Learning for Data Preparation: Learn2Clean with Human-In-The-Loop
- Design-Based Theory for Lasso Adjustment in Randomized Block Experiments and Rerandomized Experiments
- Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing
- AutoML Two-Sample Test