In traditional statistics, we are taught that uncertainty arises because we only see a small sample of a much larger, often infinite, population. If we want to know the average height of people in a country, we measure 1,000 people and use the variance of that sample to estimate how far our mean might be from the "true" population mean.
But what happens if you have data on the entire population? If you are studying the impact of a policy across all 50 U.S. states, or analyzing every transaction on a specific e-commerce platform, your "sample" is the population. Under the traditional framework, your standard errors should be zero. Yet, intuitively, we know that if we ran the experiment again, the results might change.
This realization has led to a shift in how researchers quantify the reliability of causal claims, moving from sampling-based uncertainty to design-based uncertainty.
The Two Sources of Randomness
To understand why standard errors persist in full-population data, we must distinguish between two different ways "randomness" enters a study. Alberto Abadie, Susan Athey, Guido Imbens, and Jeffrey M. Wooldridge formalize this in Sampling-based vs. Design-based Uncertainty in Regression Analysis.
- Sampling Uncertainty: The randomness of who is in your study. This is the focus of classical statistics.
- Design (or Assignment) Uncertainty: The randomness of which units received the treatment. Even if the population is fixed, the coin flips that decided which state got a tax credit and which didn't could have landed differently.
In Finite Population Causal Standard Errors, the authors argue that for causal inference, the "missingness" of counterfactuals is the primary source of error. We see state A under a policy, but we never see state A without that policy at the same moment in time. This unobserved potential outcome is a stochastic variable. Because we only observe one realization of the world, we face uncertainty about the causal effect even if we have data on every unit in existence.
Why Traditional Standard Errors Are Often Too Large
Most statistical software defaults to "robust" or "Huber-White" standard errors. These assume an infinite population. However, Abadie et al. demonstrate that when the sample is a large fraction of the population, these traditional errors are often overly conservative.
If you are interested in the descriptive effect (what happened in these specific 50 states), the sampling variance is zero. If you are interested in the causal effect (what would happen if we applied this policy elsewhere), the variance is non-zero but still typically smaller than what infinite-population formulas suggest. By accounting for the specific proportion of the population being studied, researchers can produce tighter, more precise confidence intervals.
The Causal Bootstrap
To handle these complexities in practice, researchers have developed the Causal Bootstrap. Unlike a standard bootstrap, which resamples rows of data to simulate new "samples" from a population, the causal bootstrap must account for the uncertainty in potential outcomes.
Konrad Menzel introduced a version of this in A Causal Bootstrap that imputes missing potential outcomes using a "worst-case" correlation. The goal is to ensure that the variance estimate remains valid even if the entire population is observed but the treatment assignments could have differed.
This approach is particularly vital in "small-N" settings, such as geo-experiments where the units are a handful of cities. In these cases, the heterogeneity between units is so high that the specific randomization of treatment (the "design") outweighs any sampling concerns. Jennifer Brennan, Sebastien Lahaie, and colleagues have extended this by using Integer Programming to numerically solve for the worst-case joint distribution of potential outcomes (Integer Programming for Generalized Causal Bootstrap Designs). This allows for valid inference in complex experimental designs (like matched pairs) where traditional formulas break down.
Integrating Confounding and Noise
A final facet of this research direction involves bridging the gap between statistical noise (variance) and systematic bias (unobserved confounding).
In many real-world scenarios, we cannot assume that treatment assignment was perfectly random. Jiajing Zheng and Alexander Franks argue in Copula-based Sensitivity Analysis for Multi-Treatment Causal Inference with Unobserved Confounding that we should not just look for a single point estimate. Instead, we should calculate "ignorance regions"—bounds that reflect both the uncertainty of the finite sample and the uncertainty of what we don't know about hidden confounders.
Similarly, Alexis Bellot and Silvia Chiappa emphasize that a comprehensive framework must integrate frequentist confidence intervals with causal bounds (Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding). By combining causal graphs with machine learning, they provide a way to estimate these intervals even in high-dimensional datasets.
Summary
The core intuition of this field is that causal uncertainty is not just sampling uncertainty. Even with "perfect" data on an entire population, the fact that we cannot observe the same unit in two states at once creates a fundamental floor of uncertainty. By shifting from an infinite-population mindset to a design-based framework, researchers can provide more honest—and often more precise—answers to the most important policy questions.