Missing data interpolation in integrative multi-cohort analysis with disparate covariate information

Ekaterina A. Smirnova, Yongqi Zhong, Rasha Alsaadawi, Ning Xu, Amii Kress, Jordan R. Kuiper, Mingyu Zhang, Kristen Lyall, Sheenas Martenies, Akram N. Alshawabkeh, Catherine M. Bulka, Carlos A. Camargo, Jaeun Choi, Elena Colicino, Anne L. Dunlop, Michael Elliott, Assiamira Ferrara, Tebeb Gebrestadik, Jiang Gui, Kylie K. Harrall, Tina V. Hartert, Barry M. Lester, Andrew W. Manigault, Justin Manjourides, Ni Yu, Rosalind J. Wright, Robert Wright, Katherine Ziegler, Bryan LauarXiv 2022 · DOI · arXiv

TL;DR

The relate framework introduces a two-stage approach to solve the "missing variable" problem in multi-cohort studies. By combining Random Forest-based clustering with recursive statistical testing, it identifies which groups of participants are similar enough to share data, allowing researchers to accurately fill in missing information without introducing the biases of "one-size-fits-all" imputation.

What Problem Does This Paper Address?

In modern epidemiology, researchers often perform "integrative analysis"—pooling data from dozens of independent studies (cohorts) to increase statistical power. However, these studies rarely collect the exact same variables. One cohort might have detailed data on maternal smoking but lack neighborhood-level pollution data, while another has the opposite.

This creates cohort-level missingness. Traditionally, researchers either throw out any cohort missing a variable (losing massive amounts of data) or "impute" the missing values by assuming every cohort in the pool follows the same underlying distribution. The latter is dangerous: if you use data from a high-income urban cohort to fill in gaps for a low-income rural cohort, your final results will be biased. The challenge is identifying which cohorts are actually "distributionally equivalent" so that data can be shared safely.

Methods

The authors propose a method called relate (REcursive muLtivariAte TEsting for cohort clustering), which operates in two distinct stages:

  1. Stage 1: Identifying Candidate Groups (The Random Forest Trick) Instead of using a standard distance metric, the authors use a Random Forest (RF) classifier to predict which cohort an individual belongs to based on their covariates. If the RF can easily tell participants apart by cohort, the cohorts are different. If the RF struggles to classify them, the cohorts are likely similar. They then use the "dissimilarity" scores from the RF to build a hierarchical clustering dendrogram (a tree-like diagram of similarity).

  2. Stage 2: Formal Recursive Testing To move from "looks similar" to "is statistically similar," the authors apply the Biswas and Ghosh (BG) test, a non-parametric test designed for high-dimensional data. Rather than testing every possible pair of cohorts (which is computationally expensive), they move up the dendrogram created in Stage 1, recursively testing and merging groups only if the null hypothesis of "same distribution" isn't rejected.

Datasets

The paper evaluates the method using both synthetic and real-world data:

  • Simulated Data: 10 cohorts with 15 covariates (a mix of normal, binary, binomial, and multinomial distributions) across five different scenarios ranging from "all cohorts are the same" to "four distinct underlying distributions."
  • ECHO (Environmental influences on Child Health Outcomes):
    • Study 1: Neighborhood-level exposures and pre-term birth (35 cohorts).
  • Study 2: Cardiometabolic pregnancy complications and autism spectrum disorder (ASD) traits (28 cohorts).

Key Insights and Ideas

  • Classification as a Similarity Metric: The use of a classifier's inability to distinguish between groups as a proxy for distributional similarity is a clever application of machine learning. It bypasses the need for complex parametric assumptions.
  • Efficiency through Recursion: Brute-force pairwise testing for 69 cohorts would require over 2,300 tests. By following the hierarchical tree, relate drastically reduces the number of tests and the computational time.
  • Design vs. Reality: A major "aha moment" in the real-world application was that cohorts with similar study designs (e.g., all "autism-enriched" cohorts) did not always have similar data distributions. This proves that manual grouping based on metadata is insufficient; you need a data-driven approach.

Key Results

  • High Accuracy: In simulations, relate correctly identified the number of distribution clusters with >90% accuracy in most scenarios, even with high levels of missing data (up to 60%).
  • Speed: The entire process (RF fitting + recursive testing) typically runs in under two minutes for a 10-cohort setup, making it highly scalable for large consortia.
  • Impact on Inference: In the ECHO pre-term birth study, the authors showed that "pooled" imputation (treating all 35 cohorts as one) produced statistically significant results for certain race and ethnicity categories that vanished when using the more precise relate groupings. This suggests that standard imputation methods may be generating "false positive" associations by ignoring cohort heterogeneity.

Limitations and Future Work

  • The MAR Assumption: The method assumes data is "Missing At Random" (MAR), meaning the reason data is missing is related to observed variables, not the missing values themselves.
  • Singleton Cohorts: Sometimes a cohort is so unique that it doesn't group with any others. In these cases, relate cannot perform imputation, and the cohort must be excluded—a "fail-safe" that protects data integrity but reduces sample size.
  • Small Study Limitations: The authors suggest the method may not be suitable for very small studies (3-4 cohorts) where the statistical tests lack the power to distinguish distributions.
  • Future Directions: The authors suggest exploring "superlearning" (combining multiple ML algorithms) to further improve the initial clustering stage.

Summary updated

Abstract

Integrative analysis of datasets generated by multiple cohorts is a widely-used approach for increasing sample size, precision of population estimators, and generalizability of analysis results in epidemiological studies. However, often each individual cohort dataset does not have all variables of interest for an integrative analysis collected as a part of an original study. Such cohort-level missingness poses methodological challenges to the integrative analysis since missing variables have traditionally: (1) been removed from the data for complete case analysis; or (2) been completed by missing data interpolation techniques using data with the same covariate distribution from other studies. In most integrative-analysis studies, neither approach is optimal as it leads to either loosing the majority of study covariates or challenges in specifying the cohorts following the same distributions. We propose a novel approach to identify the studies with same distributions that could be used for completing the cohort-level missing information. Our methodology relies on (1) identifying sub-groups of cohorts with similar covariate distributions using cohort identity random forest prediction models followed by clustering; and then (2) applying a recursive pairwise distribution test for high dimensional data to these sub-groups. Extensive simulation studies show that cohorts with the same distribution are correctly grouped together in almost all simulation settings. Our methods' application to two ECHO-wide Cohort Studies reveals that the cohorts grouped together reflect the similarities in study design. The methods are implemented in R software package relate.
BibTeX
@misc{smirnova2022missing,
  author = {Ekaterina A. Smirnova and Yongqi Zhong and Rasha Alsaadawi and Ning Xu and Amii Kress and Jordan R. Kuiper and Mingyu Zhang and Kristen Lyall and Sheenas Martenies and Akram N. Alshawabkeh and Catherine M. Bulka and Carlos A. Camargo and Jaeun Choi and Elena Colicino and Anne L. Dunlop and Michael Elliott and Assiamira Ferrara and Tebeb Gebrestadik and Jiang Gui and Kylie K. Harrall and Tina V. Hartert and Barry M. Lester and Andrew W. Manigault and Justin Manjourides and Ni Yu and Rosalind J. Wright and Robert Wright and Katherine Ziegler and Bryan Lau},
  title = {{Missing data interpolation in integrative multi-cohort analysis with disparate covariate information}},
  year = {2022},
  eprint = {2211.00407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.00407},
}
Figure