Population interference in panel experiments

Kevin Han, Guillaume Basse, Iavor BojinovJournal of Econometrics 2021

Introduction

In the standard framework of randomized controlled trials (RCTs), researchers typically rely on the Stable Unit Treatment Value Assumption (SUTVA). This assumption posits that the potential outcome of any given unit is unaffected by the treatment assignments of others. However, in interconnected environments—such as social networks, digital marketplaces, or advertising platforms—this assumption is frequently violated. When the treatment of one individual influences the behavior of another, the resulting population interference can lead to significant bias in causal estimates.

A recent study by researchers from Stanford and Harvard, titled "Population Interference in Panel Experiments," addresses this challenge by examining interference within the context of panel data—experiments where units are observed over multiple time steps. The paper introduces a rigorous framework for causal inference that leverages the temporal dimension to overcome the complexities of unit-to-unit interference.

The Dual Challenge: Population and Temporal Interference

Causal inference literature often treats two types of interference in isolation:

  1. Population Interference: Spillovers between units at a single point in time (e.g., a discount given to one user affecting the purchasing decisions of their peers).
  2. Temporal Interference (Carryover): The effect of a treatment administered at time $t$ persisting to influence outcomes at time $t+1$ or beyond.

In modern experimental settings, such as those managed by DoorDash, Lyft, or Meta, both forms of interference often co-occur. This paper investigates whether the addition of a temporal dimension ($T$) simplifies or complicates the estimation of treatment effects under population interference.

Methodological Framework

The authors adopt a design-based (randomization-based) inference framework. In this approach, potential outcomes are considered fixed, and the only source of randomness is the treatment assignment mechanism itself. To make the problem tractable, the authors introduce several key innovations:

Exposure Mappings and Locally Effective Assignments

To manage the exponential number of possible treatment combinations in a network, the authors utilize exposure mappings. These mappings reduce complex global treatment vectors into a manageable set of "effective treatments" based on a unit's local neighborhood. They further define Locally Effective Assignments (LEA), an assumption that a unit’s outcome is determined solely by the recent treatment history of its immediate network neighbors.

The "Blessing of Dimensionality"

One of the paper's most significant theoretical contributions is the demonstration of a "blessing of dimensionality." The authors prove three new Central Limit Theorems (CLTs) for the Average Temporal Exposure Contrast (ATEC). These theorems show that while dense interference usually makes inference difficult, a large number of time steps ($T$) can compensate for a large number of units ($n$). Specifically, if there are no carryover effects, increasing the duration of an experiment allows for valid statistical inference even in networks with dense interference that would otherwise be impossible to analyze in a cross-sectional study.

Practical Estimation: The Shrinkage Estimator

Beyond theoretical proofs, the paper introduces a novel Shrinkage Estimator designed to improve the precision of causal estimates. By introducing a "weak stability" assumption—the idea that potential outcomes do not fluctuate wildly from one day to the next—the researchers developed a convex combination estimator. This tool "borrows" information from previous time steps to reduce the variance of current estimates, effectively trading a negligible amount of bias for a significant gain in statistical power.

Empirical Validation

The authors validated their framework using a combination of synthetic and real-world datasets:

  • Social Network Simulations: Using Erdős-Rényi models, the authors confirmed that as $T$ increases, the distribution of the ATEC estimator converges to a normal distribution, even in relatively dense networks.
  • Rational Cooperation Data: Analyzing a panel experiment from Andreoni & Samuelson (2006) involving Prisoners’ Dilemma games, the authors demonstrated that their shrinkage estimator reduced the variance of treatment effect estimates by nearly 50% compared to the standard Horvitz-Thompson estimator.
  • Facebook 100 Dataset: Using the Swarthmore College social network (1,657 units, 61,049 edges), the authors showed that while standard estimators fail at low $T$ due to interference density, the "blessing of time" allows the estimator to become accurate and normally distributed as $T$ reaches 100.

Limitations and Future Directions

While the paper offers a powerful new toolkit, it notes several constraints:

  • The Carryover Penalty: The benefits of the temporal dimension are significantly diminished if carryover effects are present. When past treatments influence future outcomes, the independence between time steps is lost, requiring much stricter assumptions for valid inference.
  • Temporal Independence: The current CLTs assume that treatment assignments are independent across time, which may not apply to "dynamic" experimental designs where assignments evolve based on previous results.
  • Compliance: The framework assumes perfect adherence to treatment assignments, leaving the complexities of non-compliance for future study.

Conclusion

The research suggests a practical shift in experimental design for networked environments: when faced with the noise of social interference, increasing the duration of the experiment ($T$) may be more effective and cost-efficient than simply increasing the number of subjects ($n$). By treating time as a buffer, researchers can achieve clear causal signals in even the most interconnected populations.

Summary updated

Abstract

The phenomenon of population interference, where a treatment assigned to one experimental unit affects another experimental unit’s outcome, has received considerable attention in standard randomized experiments. The complications produced by population interference in this setting are now readily recognized, and partial remedies are well known. Less understood is the impact of population interference in panel experiments where treatment is sequentially randomized in the population, and the outcomes are observed at each time step. This paper proposes a general framework for studying population interference in panel experiments and presents new finite population estimation and inference results. Our findings suggest that, under mild assumptions, the addition of a temporal dimension to an experiment alleviates some of the challenges of population interference for certain estimands. In contrast, we show that the presence of carryover effects — that is, when past treatments may affect future outcomes — exacerbates the problem. Our results are illustrated through both an empirical analysis and an extensive simulation study.
BibTeX
@article{han2021population,
  author = {Kevin Han and Guillaume Basse and Iavor Bojinov},
  title = {{Population interference in panel experiments}},
  journal = {Journal of Econometrics},
  year = {2021},
}
Figure