KARAJ: An Efficient Adaptive Multi-Processor Tool to Streamline Genomic and Transcriptomic Sequence Data Acquisition

Mahdieh Labani, Amin Beheshti, Nigel H. Lovell, Hamid Alinejad-Rokny, Ali AfrasiabiInternational Journal of Molecular Sciences 2022 · DOI

TL;DR

Researchers often struggle with the time-consuming and complex process of finding and downloading large genomic and transcriptomic datasets from various public repositories. KARAJ is a new, efficient, and flexible Linux command-line tool that automates this entire workflow, from identifying relevant datasets within scientific articles to downloading them rapidly using a parallel processing framework.

What Problem Does This Paper Address?

The explosion of genomic and transcriptomic data presents an incredible opportunity for biological research, but it also creates a significant bottleneck: accessing and managing this data. Researchers need to sift through numerous publications to find relevant datasets, often identified by accession numbers. These datasets are frequently massive, requiring careful consideration of local storage space. Furthermore, obtaining supplementary data tables, which are crucial for downstream analysis, can be a separate, manual process. Existing tools often address only parts of this workflow, are not fully automated, or lack efficient downloading capabilities, leading to time-consuming and technically challenging data acquisition.

Methods

KARAJ is a bash-scripted command-line tool designed for Linux environments. Its core functionality involves several automated steps:

  • Input Flexibility: It accepts various inputs, including PubMed Central (PMC) IDs, publication URLs, and direct accession numbers (e.g., SRA, GEO, DDBJ, ENA, ENCODE).
  • Dataset Discovery: For PMCIDs or URLs, KARAJ uses the Lynx package to mine the article text and extract relevant accession numbers.
  • Dataset Summarization: It generates a summary of accessible datasets, including sample information and experimental design, allowing users to select specific datasets of interest.
  • Storage Management: KARAJ intelligently calculates the size of the selected datasets and checks for sufficient local disk space before initiating downloads, preventing failed downloads due to insufficient storage.
  • Metadata Generation: It creates a metadata table containing crucial information about the samples and experimental design.
  • Supplementary Data Retrieval: KARAJ can download supplementary data tables attached to publications.
  • Efficient Downloading: The tool leverages the ffq package to retrieve download links and utilizes IBM Aspera Connect, a high-speed transfer protocol, for downloading. It implements a parallel downloading framework, powered by Aspera and axel, to significantly reduce download times by utilizing multiple local system cores.

Datasets

The paper does not explicitly list datasets used for training KARAJ. However, it describes the evaluation of KARAJ's performance using publicly available datasets, including:

  • GSE126379: Used to evaluate the speedup achieved by KARAJ's parallel downloading framework compared to standard tools like wget and curl.
  • GSE126379 accession number: Used to demonstrate the retrieval of metadata tables.
  • Data generated by MacParland, et al. [36]: A large (277 GB) dataset used to test KARAJ's memory check module.

KARAJ is designed to work with data from major public repositories such as SRA, GEO, DDBJ, ENA, and ENCODE, which store data in formats like FASTQ, FASTA, BAM, SAM, GFF, GTF, and VCF.

Key Insights and Ideas

  • End-to-End Automation is Crucial: The paper highlights that the sheer volume of genomic data necessitates tools that automate the entire data acquisition pipeline, not just individual steps.
  • Storage Awareness Prevents Wasted Effort: A significant practical challenge is managing large datasets and ensuring sufficient local storage. KARAJ's built-in storage check is a valuable feature that prevents wasted download time and effort.
  • Parallel Downloading is Key for Speed: The adoption of Aspera Connect and a parallel framework is a critical innovation for drastically reducing the time required to download massive sequence files.
  • Integrated Metadata and Supplementary Data: Providing metadata and supplementary data alongside raw sequence files streamlines downstream analysis by offering context and essential supporting information.

Key Results

KARAJ demonstrated significant improvements in data acquisition efficiency:

  • Reduced Downloading Time: The parallel downloading framework powered by Aspera Connect reduced the downloading time of GSE126379 sequence data by 3.6-fold using 8 cores compared to Aspera alone.
  • Comprehensive Functionality: KARAJ automates four key tasks: dataset summarization and selection, storage space checking, metadata table generation, and supplementary data download.
  • Flexibility: The tool supports multiple input types (PMCIDs, URLs, accession numbers) and various data formats.

Limitations and Future Work

  • Linux Command-Line Interface: KARAJ is a command-line tool, which might present a learning curve for users less familiar with the Linux environment.
  • Dependency on External Tools: KARAJ relies on several external packages (e.g., ffq, pysradb, Lynx, Aspera Connect, axel, wget, Entrez Direct). While the installer script helps manage these, their availability and compatibility are crucial.
  • Future Work: The paper does not explicitly detail future work, but the established framework suggests potential for expanding support for more data types, databases, or integrating with other bioinformatics analysis pipelines. The current implementation focuses on data acquisition, leaving downstream analysis as a separate user responsibility.

Summary updated

Abstract

Here we developed <i>KARAJ</i>, a fast and flexible Linux command-line tool to automate the end-to-end process of querying and downloading a wide range of genomic and transcriptomic sequence data types. The input to KARAJ is a list of PMCIDs or publication URLs or various types of accession numbers to automate four tasks as follows; firstly, it provides a summary list of accessible datasets generated by or used in these scientific articles, enabling users to select appropriate datasets; secondly, <i>KARAJ</i> calculates the size of files that users want to download and confirms the availability of adequate space on the local disk; thirdly, it generates a metadata table containing sample information and the experimental design of the corresponding study; and lastly, it enables users to download supplementary data tables attached to publications. Further, <i>KARAJ</i> provides a parallel downloading framework powered by <i>Aspera connect</i> which reduces the downloading time significantly.
BibTeX
@article{labani2022karaj,
  author = {Mahdieh Labani and Amin Beheshti and Nigel H. Lovell and Hamid Alinejad-Rokny and Ali Afrasiabi},
  title = {{KARAJ: An Efficient Adaptive Multi-Processor Tool to Streamline Genomic and Transcriptomic Sequence Data Acquisition}},
  journal = {International Journal of Molecular Sciences},
  year = {2022},
  doi = {10.3390/ijms232214418},
}
Figure