Abstract
There is a movement in design of experiments away from the classic randomization put forward by Fisher, Cochran and others to one based on optimization. In fixed-sample trials comparing two groups, measurements of subjects are known in advance and subjects can be divided optimally into two groups based on a criterion of homogeneity or "imbalance" between the two groups. These designs are far from random. This paper seeks to understand the benefits and the costs over classic randomization in the context of different performance criterions such as Efron's worst-case analysis. In the criterion that we motivate, randomization beats optimization. However, the optimal design is shown to lie between these two extremes. Much-needed further work will provide a procedure to find this optimal designs in different scenarios in practice. Until then, it is best to randomize.
Introduction
In clinical trials and experimental research, the primary goal is to ensure that treatment and control groups are as comparable as possible. Historically, complete randomization has been the "gold standard," relying on chance to balance both known and unknown covariates. However, modern experimental design is increasingly shifting toward optimization. When subject characteristics (e.g., age, weight, medical history) are known before a trial begins, researchers can use optimization algorithms to create groups with near-perfect balance on these observed traits.
While optimization promises greater precision, it introduces a significant risk: by forcing a perfect balance on what is visible, a design may inadvertently create a "ruinous" imbalance in unobserved factors. This paper explores the fundamental trade-off between these two paradigms and proposes a framework for finding a middle ground.
Methodological Framework
The authors evaluate experimental designs using an additive treatment effect model ($y = \beta_T w + f(x) + z$) and three distinct performance metrics:
- Worst-Case Criterion (Efron’s Bound): Minimizing the error for the single most adversarial unobserved covariate.
- Mean Criterion: Minimizing the average error across all possible unobserved covariates.
- Tail Criterion: The authors' proposed metric, which minimizes a high quantile (e.g., the 95th percentile) of the error distribution. This protects against "unlucky" assignments without being as overly conservative as the worst-case approach.
To test these criteria, the study employs synthetic simulations across small ($n=20$) and large ($n=200$) samples, alongside adversarial scenarios specifically designed to challenge optimized models.
The Optimization Trap and Randomness as Regularization
A key finding is that randomization acts as a form of "regularization." By spreading the assignment across many possible vectors, it prevents the design from becoming overly sensitive to any single unobserved direction in the data. In contrast, pure optimization often suffers from a "long right tail" in its error distribution; while it performs well on average, its failures due to unobserved factors are often catastrophic.
This phenomenon illustrates a "No Free Lunch Theorem" for experimental design: any restriction of the sample space to balance observed covariates can inadvertently trigger higher variance when adversarial unobserved covariates are present.
Key Results and Adversarial Analysis
The research demonstrates that the "best" design is highly dependent on the researcher's objective. If the goal is average performance, optimization is superior. However, if the goal is to avoid the absolute worst-case scenario, randomization is more robust.
In an adversarial example provided by the authors, complete randomization (CRFB) was compared against restricted matching. The general metrics for these designs are shown in Table 1.
When specific values were applied to this case (with $m=3$), the results highlighted the potential failure of optimization. While the restricted matching design achieved a perfect 0.00 mean observed imbalance, its mean unobserved imbalance was significantly higher (3.00) than that of the random design (1.80). Consequently, the Mean Squared Error (MSE) for the treatment estimator was higher for the "optimized" design (0.75) than for the random one (0.58).
Conclusion
The study suggests that Pairwise Matching—pairing similar subjects and then randomizing within those pairs—often serves as the most practical middle ground. It significantly reduces observed imbalance while retaining enough randomness to protect against unobserved noise.
While the paper proves that a "harmonized" design exists between the extremes of pure optimization and pure randomization, a general turn-key algorithm for practitioners remains a subject for future work. Until such procedures are established, the authors conclude that maintaining randomization is the safest course of action for most experimental contexts.
Summary updated