Toward whole-genome inference of polygenic scores with fast and memory-efficient algorithms

Shadi Zabad, Chirayu Anant Haryan, Simon Gravel, Sanchit Misra, Yue LiThe American Journal of Human Genetics 2025 · DOI

TL;DR

This paper introduces a significantly faster and more memory-efficient algorithm for calculating polygenic risk scores (PRSs) from massive genomic datasets. By developing novel data structures for linkage disequilibrium (LD) matrices and optimizing inference algorithms, they enable whole-genome PRS calculations on tens of millions of variants, drastically reducing computational costs and making PRSs more accessible.

What Problem Does This Paper Address?

Genome-wide association studies (GWASs) have identified millions of genetic variants associated with various traits and diseases. However, translating these individual associations into a single, comprehensive "polygenic risk score" (PRS) that captures an individual's overall genetic predisposition has been computationally challenging. Existing methods often struggle with the sheer volume of genetic data, either by simplifying the problem (e.g., using only a small subset of variants) or by requiring excessive computational resources (time and memory). This limits the ability to perform accurate and scalable PRS inference, especially as sequencing and imputation technologies continue to generate ever-larger datasets.

Methods

The core of this work lies in two main areas:

  • Data Structures for LD Matrices:
    • Compression: They introduce a highly compressed format for Linkage Disequilibrium (LD) matrices. This involves:
  • Storing only the upper triangular portion of the symmetric LD matrix.
  • Quantizing matrix entries to lower-precision floating-point or integer types (e.g., int8 reduces storage by 8x).
  • Utilizing a cloud-native Compressed Sparse Row (CSR) format (Zarr) for efficient storage and access.
  • Numerical Stability: They address numerical instabilities in PRS inference by adding a penalty term derived from the smallest eigenvalue of the LD matrix. This helps constrain parameter estimates and prevent exploding values, particularly in high-dimensional settings.
  • Algorithmic Improvements:
    • Memory-Efficient Inference: They developed a "low-memory" or "triangular LD mode" for their variational Bayesian inference algorithm (VIPRS). This allows the algorithm to operate directly on the compressed, upper-triangular LD matrix, significantly reducing memory footprint.
  • Parallelism: They implemented two layers of parallelism:
  • Across Chromosomes: Using a process pool to analyze chromosomes independently.
  • Within Chromosomes: Parallelizing the coordinate-ascent updates using OpenMP threads.
  • Dequantize-on-the-Fly (DQF): An option to dequantize LD matrix entries only when needed during computation, further reducing memory usage at a slight runtime cost.

Datasets

  • UK Biobank: Used extensively for training and evaluating PRS models. They utilized summary statistics for 75 heritable, continuous phenotypes.
  • CARTAGENE Biobank: Used for cross-biobank validation to assess the transferability of PRS models.
  • HapMap3+ reference panel: A standard reference panel for LD information, used in various sizes (e.g., ~1.1 million variants, ~13.5 million variants, ~18 million variants).
  • LD matrices: Pre-computed LD matrices for six ancestry groups were generated using the Pan-UKB genotype data.

Key Insights and Ideas

  • LD Matrix Compression is Crucial: The paper strongly emphasizes that the way LD matrices are stored and compressed is a major bottleneck. Their novel format achieves dramatic storage reductions (up to 50x), making it feasible to handle much larger variant sets.
  • Numerical Stability is Not Just About Data Quality: While data quality is important, the spectral properties of the LD matrix itself (specifically negative eigenvalues) are a significant source of numerical instability in PRS methods. Their proposed solution directly addresses this.
  • Parallelism and Memory Efficiency are Synergistic: Combining multi-threading within the inference algorithm with memory-efficient data structures (like the compressed LD matrix) leads to substantial gains in both speed and memory usage.
  • Dense Variant Sets Offer Marginal Gains (Sometimes): While using more variants generally improves accuracy, the gains are often modest (3-5%) and can be overshadowed by model assumptions or the quality of input data, especially for well-established methods.
  • Cloud-Native Storage is the Future: Their adoption of Zarr for LD matrix storage highlights the trend towards cloud-native solutions for large-scale genomics data.

Key Results

  • Storage Reduction: Over 50-fold reduction in LD matrix storage requirements using their compressed format (e.g., int8 quantization).
  • Speedup: Orders-of-magnitude improvements in runtime and memory efficiency. VIPRS v.0.1 is up to 80 times faster than popular baselines and uses significantly less memory.
  • Scalability: The updated VIPRS can perform variational Bayesian regression on up to 18 million variants in under an hour, using less than 15 GB of RAM.
  • Accuracy: Achieved prediction accuracy comparable to state-of-the-art tools, even with massive datasets.
  • Numerical Stability: The method demonstrates stable convergence even with ultra-high dimensional data.

Limitations and Future Work

  • Hyperparameter Tuning Cost: While grid search improves accuracy, it significantly increases runtime (3-4 fold). They suggest future work could explore more efficient tuning methods.
  • Cross-Population Transferability: While their models perform well, the gains from using denser variant sets don't always translate to better cross-ancestry prediction. This suggests that factors beyond variant density, like genetic architecture differences, play a significant role.
  • Functional Annotations: The paper notes that incorporating functional genomic annotations could further improve PRS accuracy, a direction they suggest for future research.
  • Generalizability of Parallelism: While their parallelization strategies are effective, they acknowledge potential issues like "staleness" in asynchronous updates, which might require further refinement.

Summary updated

Abstract

With improved whole-genome sequencing and variant imputation techniques, modern genome-wide association studies (GWASs) have enriched our understanding of the landscape of genetic associations for thousands of disease phenotypes. However, translating the marginal associations for millions of genetic variants to integrated polygenic risk scores (PRSs) that capture their joint effects on the phenotype remains a major challenge. Due to technical and statistical constraints, commonly used PRS methods in this setting either perform heuristic pruning and thresholding or overlook most genetic association signals by restricting inference to small variant sets, such as HapMap3. Here, we present a set of algorithmic improvements and compact data structures that enable scaling summary-statistics-based PRS inference to tens of millions of variants while avoiding numerical instabilities common in such high-dimensional settings. These enhancements consist of highly compressed linkage-disequilibrium (LD) matrix format, which integrates with streamlined and parallel coordinate-ascent updating schemes. When incorporated into our existing PRS method (VIPRS), the proposed algorithms yield over 50-fold reductions in storage requirements and lead to orders-of-magnitude improvements in runtime and memory efficiency. The updated VIPRS software can now perform variational Bayesian regression over 1.1 million HapMap3 variants in under a minute. Using this scalable implementation, we applied VIPRS to 75 of the most heritable, continuous phenotypes in the UK Biobank, leveraging marginal associations for up to 18 million bi-allelic variants. These experiments demonstrated that VIPRS is 1-2 orders of magnitude more efficient than popular baselines while being competitive with the best-performing methods in terms of prediction accuracy.
BibTeX
@article{zabad2025toward,
  author = {Shadi Zabad and Chirayu Anant Haryan and Simon Gravel and Sanchit Misra and Yue Li},
  title = {{Toward whole-genome inference of polygenic scores with fast and memory-efficient algorithms}},
  journal = {The American Journal of Human Genetics},
  year = {2025},
  doi = {10.1016/j.ajhg.2025.05.002},
}
Figure