Evaluation of polygenic scoring methods in five biobanks reveals greater variability between biobanks than between methods and highlights benefits of ensemble learning

Remo Monti, Lisa Eick, Georgi Hudjashov, Kristi Läll, Stavroula Kanoni, Brooke N. Wolford, Benjamin Wingfield, Oliver Pain, Sophie Wharrie, Bradley Jermy, Aoife McMahon, Tuomo Hartonen, Henrike Heyne, Nina Mars, Kristian Hveem, Michael Inouye, David A. van Heel, Reedik Mägi, Pekka Marttinen, Samuli Ripatti, Andrea Ganna, Christoph LippertmedRxiv 2023 · DOI

Abstract

Methods to estimate polygenic scores (PGS) from genome-wide association studies are increasingly utilized. However, independent method evaluation is lacking, and method comparisons are often limited. Here, we evaluate polygenic scores derived using seven methods in five biobank studies (totaling about 1.2 million participants) across 16 diseases and quantitative traits, building on a reference-standardized framework. We conducted meta-analyses to quantify the effects of method choice, hyperparameter tuning, method ensembling and target biobank on PGS performance. We found that no single method consistently outperformed all others. PGS effect sizes were more variable between biobanks than between methods within biobanks when methods were well-tuned. Differences between methods were largest for the two investigated autoimmune diseases, seropositive rheumatoid arthritis and type 1 diabetes. For most methods, cross-validation was more reliable for tuning hyperparameters than automatic tuning. For a given target phenotype, elastic net models combining PGS across methods (ensemble PGS) tuned in the UK Biobank provided consistent, high, and cross-biobank transferable performance, increasing PGS effect sizes (β-coefficients) by a median of 5.0% relative to LDpred2 and MegaPRS. Our results provide a rich resource and reference for the analysis of polygenic scoring methods across biobanks.

Introduction

Polygenic Scores (PGS) represent a cornerstone of personalized medicine, offering the potential to predict an individual's genetic predisposition for complex diseases such as type 2 diabetes, breast cancer, or stroke. By aggregating the effects of thousands of genetic variants, these scores provide a "weather forecast" for health. However, the field faces a significant challenge: while dozens of algorithms exist to calculate these scores, they are rarely subjected to rigorous, independent benchmarking across diverse, real-world datasets.

Most method evaluations are conducted by the algorithm's authors, often leading to claims of superiority that may not hold up in different clinical contexts. This study addresses a fundamental question in genomic medicine: when a PGS fails to generalize, is the fault in the algorithm, or is it due to fundamental differences in the data to which it is applied?

Methodology and Scale

To provide a definitive "stress test" for PGS methods, researchers developed prspipe, an open-source Snakemake workflow designed for standardized evaluation. The study evaluated seven popular PGS methods—including LDpred2, MegaPRS, and lassosum—across 16 different traits.

Datasets

The scale of the study is unprecedented, involving a meta-analysis of approximately 1.2 million participants across five major global biobanks:

  • UK Biobank (UKBB)
  • FinnGen (Finland)
  • Estonian Biobank (EBB)
  • HUNT (Norway)
  • Genes & Health (GNH) (UK-based South Asian ancestry)

The analysis focused on two primary ancestry groups: European (EUR) and South Asian (SAS). The 16 traits investigated included 10 binary diseases (e.g., Type 2 Diabetes, Stroke) and 6 quantitative traits (e.g., BMI, Height).

Tuning and Ensembling

A critical technical component involved comparing automatic tuning—where algorithms estimate parameters based solely on summary statistics—against cross-validation (CV) tuning, which utilizes a subset of target biobank data to refine the model. Furthermore, the team tested an ensemble PGS approach: an elastic net model that learns to optimally weight the outputs of all seven individual methods to generate a single, more robust prediction.

Key Insights

The "Biobank Effect"

One of the most striking findings was that once methods are properly tuned, the variability in performance between biobanks was significantly higher than the variability between methods within a single biobank. This suggests that differences in clinical phenotyping—how doctors diagnose patients or how biobanks record data—represent a larger hurdle for genomic medicine than the mathematical nuances of the algorithms themselves.

The HLA Region Challenge

For autoimmune conditions such as Type 1 Diabetes and Rheumatoid Arthritis, the methods showed significant disagreement. This variability is likely driven by the HLA region on chromosome 6, a genetically complex area that remains notoriously difficult to model accurately across different scoring methods.

The Necessity of Cross-Validation

The study found that automatic tuning was generally less reliable than cross-validation. For researchers and clinicians, the takeaway is clear: if even a small amount of target data is available, using it for hyperparameter tuning is essential for maximizing predictive accuracy.

The Power of Ensembles

While specific methods might excel for individual traits (e.g., one method winning for "Height" and another for "Stroke"), the ensemble approach proved to be the most consistent "all-rounder." It provided high performance that transferred effectively across different biobanks and ancestry groups.

Results and Performance Gains

  • Effect Size Increase: The ensemble PGS increased effect sizes (β-coefficients) by a median of 5.0% compared to the two best-performing single methods, LDpred2 and MegaPRS.
  • Variance Explained: In terms of variance explained ($r^2$), the ensemble method demonstrated even more substantial gains, with median increases of 7.4% to 9% for binary traits and 10.7% to 16.1% for continuous traits.
  • Robustness: The ensemble model consistently ranked at the top for nearly all traits across both European and South Asian ancestries, highlighting its utility for cross-biobank applications.

Limitations and Future Directions

Despite the inclusion of South Asian data from Genes & Health, the study remains heavily weighted toward European ancestries, reflecting a systemic bias in genomic research. Additionally, as new algorithms are constantly released, static benchmarks can quickly become outdated.

Future research should focus on improving the modeling of complex regions like the HLA and developing decentralized benchmarking platforms where new methods can be continuously evaluated. The authors also suggest that "MultiPGS" strategies—incorporating genetic data from multiple related traits—could further enhance the predictive power of ensemble models.

Resource Availability: The full suite of results is interactively browsable at methodscomparison.intervenegeneticscores.org. The open-source workflow is available at github.com/intervene-EU-H2020/prspipe.

Summary updated

Abstract

Abstract Methods to estimate polygenic scores (PGS) from genome-wide association studies are increasingly utilized. However, independent method evaluation is lacking, and method comparisons are often limited. Here, we evaluate polygenic scores derived using seven methods in five biobank studies (totaling about 1.2 million participants) across 16 diseases and quantitative traits, building on a reference-standardized framework. We conducted meta-analyses to quantify the effects of method choice, hyperparameter tuning, method ensembling and target biobank on PGS performance. We found that no single method consistently outperformed all others. PGS effect sizes were more variable between biobanks than between methods within biobanks when methods were well-tuned. Differences between methods were largest for the two investigated autoimmune diseases, seropositive rheumatoid arthritis and type 1 diabetes. For most methods, cross-validation was more reliable for tuning hyperparameters than automatic tuning (without the use of target data). For a given target phenotype, elastic net models combining PGS across methods (ensemble PGS) tuned in the UK Biobank provided consistent, high, and cross-biobank transferable performance, increasing PGS effect sizes (β-coefficients) by a median of 5.0% relative to LDpred2 and MegaPRS (the two best performing single methods when tuned with cross-validation). Our interactively browsable online-results ( https://methodscomparison.intervenegeneticscores.org/ ) and open-source workflow prspipe ( https://github.com/intervene-EU-H2020/prspipe ) provide a rich resource and reference for the analysis of polygenic scoring methods across biobanks.
BibTeX
@misc{monti2023evaluation,
  author = {Remo Monti and Lisa Eick and Georgi Hudjashov and Kristi Läll and Stavroula Kanoni and Brooke N. Wolford and Benjamin Wingfield and Oliver Pain and Sophie Wharrie and Bradley Jermy and Aoife McMahon and Tuomo Hartonen and Henrike Heyne and Nina Mars and Kristian Hveem and Michael Inouye and David A. van Heel and Reedik Mägi and Pekka Marttinen and Samuli Ripatti and Andrea Ganna and Christoph Lippert},
  title = {{Evaluation of polygenic scoring methods in five biobanks reveals greater variability between biobanks than between methods and highlights benefits of ensemble learning}},
  year = {2023},
  doi = {10.1101/2023.11.20.23298215},
  howpublished = {medRxiv},
}
Figure