To weight or not to weight? Studying the effect of selection bias in three large EHR-linked biobanks

Maxwell Salvatore, Ritoban Kundu, Xu Shi, Christopher R. Friese, Seunggeun Lee, Lars G. Fritsche, Alison M. Mondul, David A. Hanauer, Celeste Leigh Pearce, Bhramar MukherjeemedRxiv 2024 · DOI

TL;DR

This study investigates whether applying statistical weights to "unrepresentative" biobank data actually changes the scientific conclusions we draw from them. While weighting is essential for accurately estimating disease prevalence and specific effect sizes, the researchers found that the "top hits" in large-scale discovery analyses (like PheWAS) remain remarkably stable even without weighting.

What Problem Does This Paper Address?

Electronic Health Record (EHR)-linked biobanks like the UK Biobank or the NIH’s All of Us are goldmines for medical research, but they have a fundamental flaw: they are not random samples of the population.

This "selection bias" manifests in different ways. The UK Biobank suffers from a "healthy volunteer" bias, while hospital-based biobanks (like the Michigan Genomics Initiative) are often skewed toward sicker patients or those undergoing specific procedures. If researchers treat these biobanks as perfectly representative of the general public, they risk the "Big Data Paradox"—becoming increasingly confident in results that are systematically wrong. This paper asks: when does this bias actually matter, and can we fix it by "weighting" the data?

Methods

The researchers compared three major biobanks: All of Us (AOU), the Michigan Genomics Initiative (MGI), and the UK Biobank (UKB).

To "fix" the bias, they used Inverse Probability (IP) weighting and Poststratification (PS) weighting. Essentially, they compared the biobank participants to "gold standard" representative surveys (like the National Health Interview Survey in the US). If a certain demographic (e.g., young healthy males) was underrepresented in the biobank, the weighting process gave those individuals more "votes" in the final analysis to balance the scales.

They tested the impact of these weights on four common tasks:

  1. Prevalence Estimation: How common is a disease?
  2. Phenome Structure: How many independent "clusters" of diseases exist (using PCA)?
  3. PheWAS (Phenome-Wide Association Study): Scanning thousands of diseases to find associations with a specific condition (in this case, colorectal cancer).
  4. Targeted Association: Measuring the specific link between biological sex and colorectal cancer.

Datasets

  • All of Us (AOU): $n=244,071$ (US-based, focused on oversampling underrepresented groups).
  • Michigan Genomics Initiative (MGI): $n=81,243$ (Hospital-based, patients undergoing anesthesia).
  • UK Biobank (UKB): $n=401,167$ (UK-based, population-level but volunteer-driven).
  • National Health Interview Survey (NHIS): Used as the representative US benchmark.
  • UK Census Microdata: Used as the representative UK benchmark.

Key Insights and Ideas

  • The "Aha" Moment on Discovery: For "untargeted" research—where you are just trying to find the strongest signals in the data—weighting might not be strictly necessary. The top 5–10 associations in a PheWAS remained largely the same whether weights were used or not.
  • Prevalence is Highly Sensitive: Raw disease prevalence in biobanks is often wildly off. In MGI (the hospital-based cohort), weighting dropped the estimated prevalence of many conditions significantly (Median Prevalence Ratio of 0.61), proving that hospital biobanks are heavily "enriched" with sick individuals.
  • The Big Data Paradox is Real: Having a massive sample size (like $n=400k$) does not protect you from bias. In fact, it can make bias more dangerous because the narrow confidence intervals make a wrong estimate look very precise.
  • Benchmarking Matters: By using colorectal cancer as a case study, the authors showed that weighting moved the biobank’s "wrong" estimates closer to the "true" estimates found in national cancer registries.

Key Results

  • Prevalence Shifts: After weighting, prevalence estimates generally decreased in the US cohorts (AOU and MGI) but slightly increased in the UKB, reflecting the different recruitment biases (sick-patient vs. healthy-volunteer).
  • Dimensionality Stability: Weighting had almost no impact on the "latent dimensionality" of the data. The number of principal components needed to explain 95% of the variance changed by only a few percentage points.
  • PheWAS Overlap: In the colorectal cancer PheWAS, the strongest associations (the "top hits") were consistent across weighted and unweighted models. However, weighting did "clean up" some results, removing or adding marginal hits.
  • Effect Size Correction: For the association between sex and colorectal cancer, unweighted MGI data provided an estimate outside the known national benchmark. Applying weights moved the estimate directly into the expected range.

Limitations and Future Work

The authors acknowledge that we can only weight for factors we actually know and measure (like age, sex, and race). If people are joining a biobank based on a hidden factor (like a specific genetic predisposition or a personality trait not captured in the EHR), weighting can't fix that bias.

Furthermore, the study suggests that "generic" weights (based on demographics) are a good start, but "outcome-specific" weights (tailored to a specific disease) might be even better. Future work should explore how these weighting strategies impact machine learning and predictive modeling, which are increasingly common uses for biobank data.

Summary updated

Abstract

EHR-linked biobanks should report recruitment and selection mechanisms and provide selection weights with defined target populations. Researchers should consider their intended estimands, specify source and target populations, and weight EHR-linked biobank analyses accordingly.
BibTeX
@misc{salvatore2024weight,
  author = {Maxwell Salvatore and Ritoban Kundu and Xu Shi and Christopher R. Friese and Seunggeun Lee and Lars G. Fritsche and Alison M. Mondul and David A. Hanauer and Celeste Leigh Pearce and Bhramar Mukherjee},
  title = {{To weight or not to weight? Studying the effect of selection bias in three large EHR-linked biobanks}},
  year = {2024},
  doi = {10.1101/2024.02.12.24302710},
  howpublished = {medRxiv},
}
Figure