TL;DR
Polygenic Risk Scores (PRS) often show inflated predictive accuracy when derived from summary statistics due to unaddressed subject overlap between discovery and replication datasets. This study demonstrates that removing overlapping individuals significantly reduces PRS performance, suggesting that current PRS estimates for complex traits like Alzheimer's disease may be overestimated.
What Problem Does This Paper Address?
Polygenic Risk Scores (PRS) are increasingly used to predict an individual's risk for complex genetic diseases and traits. Typically, PRS are developed using "discovery" datasets and then validated on independent "replication" datasets. A crucial assumption for valid validation is that these datasets are independent, meaning they do not share any individuals. However, when PRS are derived from publicly available summary statistics (e.g., from large Genome-Wide Association Studies or GWAS), it's difficult to ascertain and control for subject overlap between the discovery and replication sets. Prior studies that strictly enforced independence reported relatively modest gains from PRS in predictive models. This paper hypothesizes that this lack of independence, particularly when using summary statistics, leads to an overestimation of PRS predictive accuracy. The authors aim to quantify this overestimation bias, especially in the context of Alzheimer's disease (AD) prediction.
Methods
The study compares two types of PRS:
- rPRS (raw PRS): Derived directly from raw genetic data where subject overlap can be identified and removed.
- sPRS (summary statistics PRS): Derived from summary statistics of GWAS, where subject overlap is not easily identifiable.
The core methodology involves comparing PRS performance metrics (like AUC and R²) in scenarios with and without overlapping subjects. They also introduce a "scale effect" analysis using high-heritability traits from the UK Biobank to estimate the upper bounds of expected PRS performance. This helps to contextualize the observed PRS gains and identify potential inflation.
Datasets
The study utilizes the following genetic datasets:
- International Genomics of Alzheimer's Project (IGAP): Used for sPRS.
- Alzheimer's Disease Sequencing Project (ADSP): Used as a discovery set for rPRS and for cross-validation.
- Accelerating Medicine Partnership - Alzheimer's Disease (AMP-AD): Used as a replication set for rPRS.
- UK Biobank: Used to derive the "scale effect" for hypertension and height, serving as benchmarks for PRS performance.
Key Insights and Ideas
- Subject overlap is a significant confounder: The presence of overlapping individuals between discovery and replication sets can dramatically inflate PRS predictive performance.
- Summary statistics mask overlap: When using summary statistics (sPRS), the overlap is hidden, making it difficult to detect and correct for the resulting overestimation bias.
- rPRS provides a more realistic baseline: PRS derived from raw data (rPRS), where overlap can be controlled, show much lower, and often non-significant, performance gains, suggesting that these are more accurate estimates of true predictive power.
- High heritability traits can bound PRS performance: Using traits like hypertension and height from large datasets like UK Biobank provides a valuable reference point for estimating the maximum plausible PRS performance, highlighting that observed gains in other traits might be inflated.
Key Results
- For AD prediction using ADSP as discovery and AMP-AD as replication, removing 432 overlapping subjects caused the predictive gain (ΔAUC) to drop from a significant 0.069 to a non-significant 0.0017. Similarly, ΔR² dropped from 0.11 to 0.0041.
- sPRS derived from IGAP showed significantly higher performance (e.g., ΔAUC of 0.051±0.013 for ADSP replication) compared to rPRS, even after accounting for dataset size differences.
- In UK Biobank, rPRS for hypertension showed a ΔAUC of 0.0036±0.0027, indicating that PRS performance for well-powered, independent datasets is generally low.
- The study suggests that sPRS results from AD databases are likely inflated due to overlapping subjects.
Limitations and Future Work
- Indirect scale effect derivation: The study acknowledges that the derivation of the "scale effect" using UK Biobank data is indirect. However, they justify this by the higher heritability of hypertension and height, and the larger sample sizes available, arguing these provide robust upper bounds for PRS performance.
- Need for careful data curation: The authors emphasize the critical need for researchers to carefully screen and filter out overlapping subjects, especially within the same ethnic group, when using summary statistics for PRS studies.
- Future work: The paper implicitly suggests future research should focus on developing methods to reliably detect and account for subject overlap when using summary statistics, and on establishing more rigorous reporting standards for PRS studies to ensure transparency and prevent overestimation.
Summary updated