Design Based Inference for Randomized Experiments

This direction is merged into another direction: network interference. Its last version stays readable and citable here.

Version 1 of 121 papers31 conceptsfrozen 2026-10-04

In traditional statistics, we often imagine that our data is a random sample drawn from an infinite "super-population." We assume there is some underlying distribution—like a Bell curve—and our goal is to estimate its parameters. However, in modern randomized controlled trials (RCTs), this assumption is often a fiction. If you are testing a new feature on all users in New York City, or analyzing a fixed cohort of students, there is no "infinite population" to sample from.

A growing body of research, championed by Susan Athey and Guido Imbens in The Econometrics of Randomized Experiments, argues for a different perspective: design-based inference. In this framework, we treat the potential outcomes of the participants as fixed, immutable constants. The only source of randomness in the entire system is the "coin flip" used to assign treatment.

The Intuition of Fixed Outcomes

To understand why this works, imagine a spreadsheet where every participant has two columns: "Outcome if Treated" and "Outcome if Control." In the design-based view, these numbers are already written in the spreadsheet before the experiment begins. They are not random variables; they are fixed facts about those specific people.

The "randomness" only enters when we decide which column to observe for each person. Because we know exactly how we flipped the coins (the assignment mechanism), we can calculate the probability of every possible arrangement of treatment and control. As Iavor Bojinov, Ashesh Rambachan, and Neil Shephard explain in Panel experiments and dynamic causal effects: A finite population perspective, this allows us to derive p-values and confidence intervals directly from the experimental design itself, rather than from assumptions about the population's distribution.

Robustness to Model Misspecification

One of the most powerful features of design-based inference is its immunity to "wrong" models. In traditional regression, if you assume a linear relationship but the true relationship is curved, your results might be biased.

In a design-based framework, the validity of the estimate doesn't depend on the model being right. Hanzhong Liu and Yuehan Yang demonstrate in Blocking, rerandomization, and regression adjustment in randomized experiments with high-dimensional covariates that even if you use complex machine learning tools like Lasso to adjust for covariates, the treatment effect remains valid. The randomness of the assignment acts as a "safety net," ensuring that even if your outcome model is misspecified, your causal claims remain statistically sound.

Anytime-Valid Inference and Adaptive Designs

In online experimentation, researchers often want to "peek" at results or use adaptive algorithms like Multi-Armed Bandits (MAB). Traditional methods fail here because the data is no longer independent and identically distributed (i.i.d.)—the probability of treatment changes based on past results.

Design-based inference solves this by leveraging the known, shifting probabilities of the assignment algorithm. Dae-Woong Ham, Iavor Bojinov, and colleagues show in Design-Based Confidence Sequences and Design-Based Inference for Multi-arm Bandits that we can create "anytime-valid" confidence sequences. These allow experimenters to stop a trial at any moment without inflating the false-positive rate, because the math accounts for the specific way the bandit chose to assign treatments.

Handling Spillovers and Interference

In complex systems like marketplaces (Uber, Amazon) or social networks, treating one person often affects another. This "interference" breaks the standard assumption that units are independent.

Design-based methods are uniquely suited for these "messy" environments. Because the inference is tied to the assignment mechanism, researchers can model how treatment "spills over" across a network or geographic area. Peter M. Aronow and Cyrus Samii have extended this to spatial experiments in Inference in Spatial Experiments with Interference, providing a way to detect "leakage" effects without needing a perfect parametric model of how people interact. Similarly, Patrick Bajari and Guido Imbens apply this to Multiple Randomization Designs to account for the competitive interference inherent in digital marketplaces.

Summary

Design-based inference shifts the focus from what the population looks like to how the experiment was run. By treating outcomes as fixed and the assignment process as the sole source of uncertainty, it provides a robust, model-free foundation for causal inference that remains valid even in the face of non-stationary data, complex interference, and adaptive experimental designs.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_design_based_inference_for_randomized_ex_6192c58a,
  title = {Design Based Inference for Randomized Experiments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/design-based-inference-for-randomized-experiments/rdr_6192c58a2c6c45709b790e1db2f05426},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_design_based_inference_for_randomized_ex_6192c58a_v1,
  title = {Design Based Inference for Randomized Experiments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/design-based-inference-for-randomized-experiments/rdr_6192c58a2c6c45709b790e1db2f05426/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}