Abstract
Methods to estimate polygenic scores (PGS) from genome-wide association studies are increasingly utilized. However, independent method evaluation is lacking, and method comparisons are often limited. Here, we evaluate polygenic scores derived using seven methods in five biobank studies (totaling about 1.2 million participants) across 16 diseases and quantitative traits, building on a reference-standardized framework. We conducted meta-analyses to quantify the effects of method choice, hyperparameter tuning, method ensembling and target biobank on PGS performance. We found that no single method consistently outperformed all others. PGS effect sizes were more variable between biobanks than between methods within biobanks when methods were well-tuned. Differences between methods were largest for the two investigated autoimmune diseases, seropositive rheumatoid arthritis and type 1 diabetes. For most methods, cross-validation was more reliable for tuning hyperparameters than automatic tuning. For a given target phenotype, elastic net models combining PGS across methods (ensemble PGS) tuned in the UK Biobank provided consistent, high, and cross-biobank transferable performance, increasing PGS effect sizes (β-coefficients) by a median of 5.0% relative to LDpred2 and MegaPRS. Our results provide a rich resource and reference for the analysis of polygenic scoring methods across biobanks.
Introduction
Polygenic Scores (PGS) represent a cornerstone of personalized medicine, offering the potential to predict an individual's genetic predisposition for complex diseases such as type 2 diabetes, breast cancer, or stroke. By aggregating the effects of thousands of genetic variants, these scores provide a "weather forecast" for health. However, the field faces a significant challenge: while dozens of algorithms exist to calculate these scores, they are rarely subjected to rigorous, independent benchmarking across diverse, real-world datasets.
Most method evaluations are conducted by the algorithm's authors, often leading to claims of superiority that may not hold up in different clinical contexts. This study addresses a fundamental question in genomic medicine: when a PGS fails to generalize, is the fault in the algorithm, or is it due to fundamental differences in the data to which it is applied?
Methodology and Scale
To provide a definitive "stress test" for PGS methods, researchers developed prspipe, an open-source Snakemake workflow designed for standardized evaluation. The study evaluated seven popular PGS methods—including LDpred2, MegaPRS, and lassosum—across 16 different traits.
Datasets
The scale of the study is unprecedented, involving a meta-analysis of approximately 1.2 million participants across five major global biobanks:
- UK Biobank (UKBB)
- FinnGen (Finland)
- Estonian Biobank (EBB)
- HUNT (Norway)
- Genes & Health (GNH) (UK-based South Asian ancestry)
The analysis focused on two primary ancestry groups: European (EUR) and South Asian (SAS). The 16 traits investigated included 10 binary diseases (e.g., Type 2 Diabetes, Stroke) and 6 quantitative traits (e.g., BMI, Height).
Tuning and Ensembling
A critical technical component involved comparing automatic tuning—where algorithms estimate parameters based solely on summary statistics—against cross-validation (CV) tuning, which utilizes a subset of target biobank data to refine the model. Furthermore, the team tested an ensemble PGS approach: an elastic net model that learns to optimally weight the outputs of all seven individual methods to generate a single, more robust prediction.
Key Insights
The "Biobank Effect"
One of the most striking findings was that once methods are properly tuned, the variability in performance between biobanks was significantly higher than the variability between methods within a single biobank. This suggests that differences in clinical phenotyping—how doctors diagnose patients or how biobanks record data—represent a larger hurdle for genomic medicine than the mathematical nuances of the algorithms themselves.
The HLA Region Challenge
For autoimmune conditions such as Type 1 Diabetes and Rheumatoid Arthritis, the methods showed significant disagreement. This variability is likely driven by the HLA region on chromosome 6, a genetically complex area that remains notoriously difficult to model accurately across different scoring methods.
The Necessity of Cross-Validation
The study found that automatic tuning was generally less reliable than cross-validation. For researchers and clinicians, the takeaway is clear: if even a small amount of target data is available, using it for hyperparameter tuning is essential for maximizing predictive accuracy.
The Power of Ensembles
While specific methods might excel for individual traits (e.g., one method winning for "Height" and another for "Stroke"), the ensemble approach proved to be the most consistent "all-rounder." It provided high performance that transferred effectively across different biobanks and ancestry groups.
Results and Performance Gains
- Effect Size Increase: The ensemble PGS increased effect sizes (β-coefficients) by a median of 5.0% compared to the two best-performing single methods, LDpred2 and MegaPRS.
- Variance Explained: In terms of variance explained ($r^2$), the ensemble method demonstrated even more substantial gains, with median increases of 7.4% to 9% for binary traits and 10.7% to 16.1% for continuous traits.
- Robustness: The ensemble model consistently ranked at the top for nearly all traits across both European and South Asian ancestries, highlighting its utility for cross-biobank applications.
Limitations and Future Directions
Despite the inclusion of South Asian data from Genes & Health, the study remains heavily weighted toward European ancestries, reflecting a systemic bias in genomic research. Additionally, as new algorithms are constantly released, static benchmarks can quickly become outdated.
Future research should focus on improving the modeling of complex regions like the HLA and developing decentralized benchmarking platforms where new methods can be continuously evaluated. The authors also suggest that "MultiPGS" strategies—incorporating genetic data from multiple related traits—could further enhance the predictive power of ensemble models.
Resource Availability: The full suite of results is interactively browsable at methodscomparison.intervenegeneticscores.org. The open-source workflow is available at github.com/intervene-EU-H2020/prspipe.
Summary updated