Abstract
Missing observations are common in cluster randomised trials (CRTs). Approaches taken to handling such missing data include complete case analysis (CCA), single-level multiple imputation (SMI) that ignores clustering, multiple imputation with a fixed effect for each cluster (FMI), and multilevel multiple imputation (MMI). We conducted a simulation study to assess the performance of these approaches in terms of confidence interval coverage and empirical bias in estimated treatment effects. Missing-at-random (MAR) clustered data scenarios were simulated following a full-factorial design. An Analysis of Variance (ANOVA) was carried out to study the influence of simulation factors on performance. When the randomised treatment arm was associated with missingness, CCA resulted in biased treatment effect estimates. Across all missing data mechanisms considered, the multiple imputation methods provided estimators with negligible bias. However, confidence interval coverage was generally in excess of nominal levels (up to 99.8%) following FMI and too low following SMI. MMI led to coverage levels of approximately 95% throughout. The approach to handling missing data was the most influential factor on bias and coverage, with the number and size of clusters and the intraclass correlation coefficient (ICC) also playing significant roles.
Introduction
In clinical and social science research, Cluster Randomised Trials (CRTs) are frequently employed to evaluate interventions applied to groups, such as schools, hospitals, or geographical areas. This structure creates hierarchical or "nested" data, where individuals within the same cluster tend to be more similar to one another than to individuals in different clusters. This similarity is quantified by the Intraclass Correlation Coefficient (ICC).
A significant challenge in CRTs is the presence of missing data. If researchers ignore missing entries through Complete Case Analysis (CCA), they risk introducing significant bias, particularly if the missingness is related to the treatment arm. While Multiple Imputation (MI) is a standard technique to address missingness by filling in values multiple times to account for uncertainty, its application to hierarchical data is not straightforward. This paper investigates the optimal MI strategy for bivariate outcomes—where two related results, such as cost and health improvement, are measured simultaneously.
Methodological Approaches
The study compared four primary strategies for handling missing data within a hierarchical framework:
- Complete Case Analysis (CCA): This method deletes any individual with missing data, effectively ignoring the missingness mechanism.
- Single-level Multiple Imputation (SMI): This approach imputes data while ignoring the cluster structure entirely, treating all individuals as independent.
- Fixed-effects Multiple Imputation (FMI): This method includes the cluster as a fixed dummy variable within the imputation model to account for group membership.
- Multilevel Multiple Imputation (MMI): This strategy utilizes a random-effects model that explicitly accounts for hierarchical variance and the shared correlation within clusters.
Simulation Design
To evaluate these methods, the authors conducted a large-scale simulation study using a full-factorial design. This allowed for the control of "ground truth" across 192 different scenarios. The synthetic datasets were characterized by:
- Sample Size: A fixed total of 500 individuals.
- Cluster Configurations: Variations included many small clusters ($J=50$), few large clusters ($J=10$), and unbalanced designs.
- Factors: The study varied the ICC, missingness mechanisms (assuming Missing At Random, or MAR), and the strength of association between variables.
Performance was measured based on empirical bias (the distance between the estimate and the true value) and confidence interval coverage (the frequency with which the calculated 95% interval contained the true value).
Key Findings
The "Goldilocks" Problem of Coverage
The study revealed a distinct hierarchy in how well each method captured uncertainty:
- SMI (Ignoring clusters): Proved too aggressive. By ignoring the shared variance within clusters, it underestimated uncertainty, leading to narrow confidence intervals and significant under-coverage (often dropping below 90%).
- FMI (Fixed effects): Proved too conservative. By treating every cluster as an independent fixed category, it overestimated uncertainty, leading to unnecessarily wide intervals and over-coverage (reaching as high as 99.8%).
- MMI (Multilevel): Proved to be "just right." By accurately modeling the shared variance, MMI consistently achieved the target 95% coverage across nearly all simulated scenarios.
Bias and Treatment-Dependent Missingness
When the reason for data missingness was associated with the treatment arm, CCA produced substantially biased estimates, with bias reaching as high as 46%. In contrast, all three Multiple Imputation methods (SMI, FMI, and MMI) successfully reduced bias to negligible levels (generally < 3.5%). However, FMI was noted as being systematically inefficient, producing wider confidence intervals than MMI even when the estimates themselves were unbiased.
The Role of the ICC
The Intraclass Correlation Coefficient was identified as a critical driver of performance. As the ICC increases—indicating higher similarity within clusters—the penalty for using an inappropriate imputation method (like SMI) becomes significantly steeper.
Conclusions and Limitations
The research demonstrates that Multilevel Multiple Imputation (MMI) is the most reliable method for handling missing bivariate outcomes in CRTs. While simpler methods like SMI or FMI are easier to implement, they fail to accurately represent the uncertainty inherent in hierarchical data.
The authors acknowledge certain boundaries to this study. The simulations assumed data was Missing At Random (MAR) and did not explore Missing Not At Random (MNAR) scenarios, where missingness is tied to the unobserved values themselves. Additionally, the study focused on missing outcomes rather than missing baseline covariates or the dropout of entire clusters. Future work should focus on developing more accessible software tools for MMI to encourage its adoption in standard trial analysis.
Summary updated