Abstract
Causal inference analyses often use existing observational data, which in many cases has some clustering of individuals. In this paper, we discuss propensity score weighting methods in a multilevel setting where within clusters individuals share unmeasured confounders that are related to treatment assignment and the potential outcomes. We focus in particular on settings where models with fixed cluster effects are either not feasible or not useful due to the presence of a large number of small clusters. We found, both through numerical experiments and theoretical derivations, that a strategy of grouping clusters with similar treatment prevalence and estimating propensity scores within such cluster groups is effective in reducing bias from unmeasured cluster-level covariates under mild conditions on the outcome model. We apply our proposed method in evaluating the effectiveness of centre-based pre-school programme participation on children’s achievement at kindergarten, using the Early Childhood Longitudinal Study Kindergarten data.
Introduction
In observational studies, researchers frequently encounter "multilevel" or "clustered" data, such as students nested within schools. A significant challenge in these settings is the "unmeasured context problem," where clusters share unobserved characteristics that influence both treatment assignment and outcomes. For instance, a school's neighborhood socioeconomic status might affect both pre-school attendance and subsequent math scores, but this data may be missing from the analysis.
Standard solutions typically involve either fully pooled models, which ignore cluster context and lead to bias, or cluster-specific (fixed effects) models, which are often infeasible when clusters are small (the "small cluster" problem). This paper introduces a "partially pooled" propensity score weighting method that provides a robust middle ground, effectively reducing bias even when individual clusters are too small for traditional fixed-effects approaches.
Methods: Partially Pooled Propensity Score Estimation
The authors propose a three-step workflow to capture cluster context without requiring large sample sizes for every cluster:
- Calculate Treatment Prevalence: Determine the proportion of individuals who received the treatment ($p_h$) for every cluster.
- Selective Grouping: Use a clustering algorithm (Partitioning Around Medoids, or PAM) to group clusters with similar treatment prevalences into a small number of "cluster groups" ($G$).
- Group-Specific Estimation: Estimate propensity scores separately within each group using random effects models.
As shown in the causal graph above, the treatment prevalence ($p_h$) acts as a proxy for unmeasured cluster-level variables ($U_h$). Clusters with similar treatment rates are likely to share similar unmeasured contextual backgrounds. By grouping them, the researchers create "pseudo-populations" large enough for stable estimation but similar enough to cancel out bias from missing variables.
Key Insights: Prevalence as a Proxy
The core innovation is the realization that one does not need to measure a confounder directly if its "footprint" is observable. If an unmeasured factor $U_h$ causes higher treatment rates in certain schools, the treatment rate itself becomes a useful signal for grouping similar schools together.
In the ECLS-K dataset, the authors demonstrated this by showing how census regions (a cluster-level covariate) vary systematically with treatment prevalence. The partially pooled method leverages this correlation to recover balance for hidden variables. Mathematically, the authors show that bias is tied to the deviation of a cluster’s treatment rate from its group's average; minimizing this deviation through smart grouping directly minimizes bias.
Key Results and Applications
- Bias Reduction: In simulations with small clusters (5–25 individuals), the partially pooled method consistently showed the lowest absolute bias compared to fully pooled models, particularly when unmeasured confounders also acted as effect modifiers.
- Improved Balance: In the ECLS-K application, the method reduced the Standardized Mean Difference (SMD) for hidden cluster-level variables from 0.211 to 0.026, effectively "recovering" balance without direct access to the data.
- Causal Impact: Applying the method to ECLS-K data, the authors estimated that center-based pre-school increases kindergarten math scores by approximately 1.84 points (roughly 0.2 standard deviations). This estimate is more conservative and likely more accurate than the unweighted raw difference of 4.36 points.
Limitations and Future Work
While effective, the method requires that each group contains both treated and control individuals (positivity). Future work should explore data-adaptive methods for choosing the optimal number of groups ($G$), which was fixed at 10 in this study. Additionally, the method specifically targets cluster-level unmeasured confounding and still assumes that all relevant individual-level confounders are observed.
Summary updated