DiGeST: Distributed Computing for Scalable Gene and Variant Ranking with Hadoop/Spark

Yann‐Aël Le Borgne, Raphaël Helaers, Tom Lenaerts, Marc Abramowicz, Guillaume Smits, Gianluca BontempibioRxiv 2017 · DOI

TL;DR

This paper introduces DiGeST, a novel distributed computing framework built on Hadoop/Spark, designed to efficiently analyze massive genomic datasets for genetic association studies. It addresses the scalability limitations of existing tools, enabling researchers to identify disease-associated variants and genes across large populations and complex genetic interactions.

What Problem Does This Paper Address?

The explosion of next-generation sequencing technologies has led to an unprecedented amount of genomic data. Clinical genomics research increasingly relies on analyzing this data to uncover genetic variants and genes associated with diseases. A common approach is Genetic Association Studies (GAS), which compare genotype frequencies between diseased and healthy individuals. However, traditional GAS tools are designed for single machines and struggle to handle the sheer volume and complexity of modern genomic datasets, especially when looking for multi-locus genetic associations (interactions between multiple genes). This creates a bottleneck in discovering the genetic underpinnings of complex diseases.

Methods

DiGeST tackles this challenge by leveraging the power of Hadoop and Spark for distributed computing. The system is comprised of two main components:

  • Hadoop/Spark Backend: This provides a robust and scalable infrastructure for storing and processing large genomic datasets. It utilizes Hadoop Distributed File System (HDFS) for data storage and Spark for distributed computation.
  • Web Frontend: A user-friendly interface built with Shiny R allows researchers to easily define filtering criteria, create case and control groups, and explore the results of their analyses.

Key methodological contributions include:

  • Efficient Data Handling: DiGeST uses the Apache Parquet format for efficient storage and querying of variant data, offering better compression and query performance than traditional formats like VCF.
  • Distributed Genotype Matrix Creation: The system can efficiently create genotype matrices from large variant databases in a distributed manner, handling sparse data effectively.
  • Scalable Scoring: DiGeST implements distributed algorithms for scoring variants, genes, and even pairs of variants/genes, allowing for the identification of both single-locus and multi-locus associations. This involves partitioning the data and processing it in parallel across multiple nodes.
  • Flexible Filtering: The web interface allows users to define complex filtering rules using a SQL-like syntax, enabling precise selection of variants and samples for analysis.

Datasets

The primary dataset used for evaluation in this paper is the 1000 Genomes Project phase 3 (1000GPp3) exome data. This dataset contains SNP data for 2504 individuals and approximately 2.7 million variants. The authors also mention using annotation fields from dbSNP, dbNSFP, and SnpEff.

Key Insights and Ideas

  • Hadoop/Spark as a Game-Changer for Genomics: The paper strongly advocates for the adoption of Hadoop/Spark in bioinformatics pipelines, demonstrating its ability to handle the scale and complexity of genomic data analysis effectively.
  • Scalability is Achievable: DiGeST shows that distributed computing frameworks can scale almost proportionally with available computing resources, significantly reducing analysis times for massive datasets.
  • Abstraction of Complexity: Spark's high-level API abstracts away much of the complexity of distributed computing, making it easier for bioinformaticians to develop and deploy scalable applications.
  • Web Interface for Accessibility: The user-friendly web interface democratizes access to powerful distributed analysis tools, making them usable by a broader range of researchers.
  • Parquet for Efficient Storage: The choice of Parquet as the data storage format is crucial for performance, enabling faster data loading and querying in a distributed environment.

Key Results

  • Scalable Performance: DiGeST demonstrated excellent scalability. For instance, creating the genotype matrix for 2.7 million variants and 2504 samples took approximately 5000 seconds with one executor, but was reduced to 150 seconds with 50 executors, showing significant speedups.
  • Efficient Scoring: Variant scoring also showed impressive speedups, with completion times reduced from around 160 seconds with one executor to 7 seconds with 50 executors for 2.7 million variants.
  • Gene and Pairwise Variant Analysis: The system successfully scaled to analyze gene rankings and pairwise variant interactions, with processing times showing linear or quadratic trends depending on the analysis type.
  • Reduced Disk Footprint: Converting the variant data to Parquet format reduced the database size to slightly less than 40GB.

Limitations and Future Work

  • Hardware Bottlenecks: The paper identifies disk access and network bandwidth as the primary bottlenecks in execution times, even with distributed computing.
  • Alternative Frameworks: While Hadoop/Spark is powerful, the authors acknowledge that High-Performance Computing (HPC) solutions and dedicated hardware like GPUs and FPGAs may still offer an edge in raw computational capacity.
  • Future Work:
    • Theoretical Analysis: Further theoretical analysis of communication and parallelism tradeoffs using models is planned.
  • Integration: Improving integration and interoperability with other bioinformatics tools is a key research direction.
  • Extended Scoring Functions: Extending DiGeST to support a wider range of statistical tests (e.g., Cochran-Armitage, Hotelling's T²) and aggregative tests (e.g., CMC, CAST) is a priority.
  • Machine Learning Integration: Investigating the integration of machine learning techniques for variant and gene prioritization within the DiGeST pipeline is a promising avenue.

Summary updated

Abstract

Abstract Background The advent of next-generation sequencing technologies has opened new avenues for clinical genomics research. In particular, as sequencing costs continue to decrease, an ever-growing number of clinical genomics institutes now rely on DNA sequencing studies at varying scales - genome, exome, mendeliome - for uncovering disease-associated variants or genes, in both rare and non-rare diseases. A common methodology for identifying such variants or genes is to rely on genetic association studies (GAS), that test whether allele or genotype frequencies differ between two groups of individuals, usually diseased subjects and healthy controls. Current bioinformatics tools for performing GAS are designed to run on standalone machines, and do not scale well with the increasing size of study designs and the search for multi-locus genetic associations. More efficient distributed and scalable data analysis solutions are needed to address this challenge. Results We developed a Big Data solution stack for distributing computations in genetic association studies, that address both single and multi-locus associations. The proposed stack, called DiGeST (Distributed Gene/variant Scoring Tool) is divided in two main components: a Hadoop/Spark high-performance computing back-end for efficient data storage and distributed computing, and a Web front-end providing users with a rich set of options to filter, compare and explore exome data from different sample populations. Using exome data from the 1000 Genomes Project, we show that our distributed implementation smoothly scales with computing resources. We make the resulting software stack Open-Source, and provide virtualisation scripts to run the complete environment both on standalone machine or Hadoop-based cluster. Conclusions Hadoop/Spark provides a powerful and well-suited distributed computing framework for genetic association studies. Our work illustrates the flexibility, ease of use and scalability of the framework, and more generally advocates for its wider adoption in bioinformatics pipelines.
BibTeX
@misc{borgne2017digest,
  author = {Yann‐Aël Le Borgne and Raphaël Helaers and Tom Lenaerts and Marc Abramowicz and Guillaume Smits and Gianluca Bontempi},
  title = {{DiGeST: Distributed Computing for Scalable Gene and Variant Ranking with Hadoop/Spark}},
  year = {2017},
  doi = {10.1101/168633},
  howpublished = {bioRxiv},
}
Figure