Overestimated Polygenic Prediction due to Overlapping Subjects in Genetic Datasets

David K. Park, Mingshen Chen, Seungsoo Kim, Yoonjung Yoonie Joo, Rebekah K. Loving, Hyoung Seop Kim, Jiook Cha, Shinjae Yoo, Jong Hun KimbioRxiv 2022 · DOI

TL;DR

Polygenic Risk Scores (PRS) often show inflated predictive accuracy when derived from summary statistics due to unaddressed subject overlap between discovery and replication datasets. This study demonstrates that removing overlapping individuals significantly reduces PRS performance, suggesting that current PRS estimates for complex traits like Alzheimer's disease may be overestimated.

What Problem Does This Paper Address?

Polygenic Risk Scores (PRS) are increasingly used to predict an individual's risk for complex genetic diseases and traits. Typically, PRS are developed using "discovery" datasets and then validated on independent "replication" datasets. A crucial assumption for valid validation is that these datasets are independent, meaning they do not share any individuals. However, when PRS are derived from publicly available summary statistics (e.g., from large Genome-Wide Association Studies or GWAS), it's difficult to ascertain and control for subject overlap between the discovery and replication sets. Prior studies that strictly enforced independence reported relatively modest gains from PRS in predictive models. This paper hypothesizes that this lack of independence, particularly when using summary statistics, leads to an overestimation of PRS predictive accuracy. The authors aim to quantify this overestimation bias, especially in the context of Alzheimer's disease (AD) prediction.

Methods

The study compares two types of PRS:

  • rPRS (raw PRS): Derived directly from raw genetic data where subject overlap can be identified and removed.
  • sPRS (summary statistics PRS): Derived from summary statistics of GWAS, where subject overlap is not easily identifiable.

The core methodology involves comparing PRS performance metrics (like AUC and R²) in scenarios with and without overlapping subjects. They also introduce a "scale effect" analysis using high-heritability traits from the UK Biobank to estimate the upper bounds of expected PRS performance. This helps to contextualize the observed PRS gains and identify potential inflation.

Datasets

The study utilizes the following genetic datasets:

  • International Genomics of Alzheimer's Project (IGAP): Used for sPRS.
  • Alzheimer's Disease Sequencing Project (ADSP): Used as a discovery set for rPRS and for cross-validation.
  • Accelerating Medicine Partnership - Alzheimer's Disease (AMP-AD): Used as a replication set for rPRS.
  • UK Biobank: Used to derive the "scale effect" for hypertension and height, serving as benchmarks for PRS performance.

Key Insights and Ideas

  • Subject overlap is a significant confounder: The presence of overlapping individuals between discovery and replication sets can dramatically inflate PRS predictive performance.
  • Summary statistics mask overlap: When using summary statistics (sPRS), the overlap is hidden, making it difficult to detect and correct for the resulting overestimation bias.
  • rPRS provides a more realistic baseline: PRS derived from raw data (rPRS), where overlap can be controlled, show much lower, and often non-significant, performance gains, suggesting that these are more accurate estimates of true predictive power.
  • High heritability traits can bound PRS performance: Using traits like hypertension and height from large datasets like UK Biobank provides a valuable reference point for estimating the maximum plausible PRS performance, highlighting that observed gains in other traits might be inflated.

Key Results

  • For AD prediction using ADSP as discovery and AMP-AD as replication, removing 432 overlapping subjects caused the predictive gain (ΔAUC) to drop from a significant 0.069 to a non-significant 0.0017. Similarly, ΔR² dropped from 0.11 to 0.0041.
  • sPRS derived from IGAP showed significantly higher performance (e.g., ΔAUC of 0.051±0.013 for ADSP replication) compared to rPRS, even after accounting for dataset size differences.
  • In UK Biobank, rPRS for hypertension showed a ΔAUC of 0.0036±0.0027, indicating that PRS performance for well-powered, independent datasets is generally low.
  • The study suggests that sPRS results from AD databases are likely inflated due to overlapping subjects.

Limitations and Future Work

  • Indirect scale effect derivation: The study acknowledges that the derivation of the "scale effect" using UK Biobank data is indirect. However, they justify this by the higher heritability of hypertension and height, and the larger sample sizes available, arguing these provide robust upper bounds for PRS performance.
  • Need for careful data curation: The authors emphasize the critical need for researchers to carefully screen and filter out overlapping subjects, especially within the same ethnic group, when using summary statistics for PRS studies.
  • Future work: The paper implicitly suggests future research should focus on developing methods to reliably detect and account for subject overlap when using summary statistics, and on establishing more rigorous reporting standards for PRS studies to ensure transparency and prevent overestimation.

Summary updated

Abstract

ABSTRACT Recently, polygenic risk score (PRS) has gained significant attention in studies involving complex genetic diseases and traits. PRS is often derived from summary statistics, from which the independence between discovery and replication sets cannot be monitored. Prior studies, in which the independence is strictly observed, report a relatively low gain from PRS in predictive models of binary traits. We hypothesize that the independence assumption may be compromised when using the summary statistics, and suspect an overestimation bias in the predictive accuracy. To demonstrate the overestimation bias in the replication dataset, prediction performances of PRS models are compared when overlapping subjects are either present or removed. We consider the task of Alzheimer’s disease (AD) prediction across genetics datasets, including the International Genomics of Alzheimer’s Project (IGAP), AD Sequencing Project (ADSP), and Accelerating Medicine Partnership - Alzheimer’s Disease (AMP-AD). PRS is computed from either sequencing studies for ADSP and AMP-AD (denoted as rPRS) or the summary statistics for IGAP (sPRS). Two variables with the high heritability in UK Biobank, hypertension, and height, are used to derive an exemplary scale effect of PRS. Based on the scale effect, the expected performance of sPRS is computed for AD prediction. Using ADSP as a discovery set for rPRS on AMP-AD, ΔAUC and ΔR 2 (performance gains in AUC and R 2 by PRS) record 0.069 and 0.11, respectively. Both drop to 0.0017 and 0.0041 once overlapping subjects are removed from AMP-AD. sPRS is derived from IGAP, which records ΔAUC and ΔR 2 of 0.051±0.013 and 0.063±0.015 for ADSP and 0.060 and 0.086 for AMP-AD, respectively. On UK Biobank, rPRS performances for hypertension assuming a similar size of discovery and replication sets are 0.0036±0.0027 (ΔAUC) and 0.0032±0.0028 (ΔR 2 ). For height, ΔR 2 is 0.029±0.0037. Considering the high heritability of hypertension and height of UK Biobank, we conclude that sPRS results from AD databases are inflated. The higher performances relative to the size of the discovery set were observed in PRS studies of several diseases. PRS performances for binary traits, such as AD and hypertension, turned out unexpectedly low. This may, along with the difference in linkage disequilibrium, explain the high variability of PRS performances in cross-nation or cross-ethnicity applications, i.e., when there are no overlapping subjects. Hence, for sPRS, potential duplications should be carefully considered within the same ethnic group.
BibTeX
@misc{park2022overestimated,
  author = {David K. Park and Mingshen Chen and Seungsoo Kim and Yoonjung Yoonie Joo and Rebekah K. Loving and Hyoung Seop Kim and Jiook Cha and Shinjae Yoo and Jong Hun Kim},
  title = {{Overestimated Polygenic Prediction due to Overlapping Subjects in Genetic Datasets}},
  year = {2022},
  doi = {10.1101/2022.01.19.476997},
  howpublished = {bioRxiv},
}

Loading related content...

Figure