“GPress: a framework for querying General Feature Format (GFF) files and feature expression files in a compressed form”

Qingxi Meng, Idoia Ochoa, Mikel HernáezbioRxiv 2019 · DOI

TL;DR

This paper introduces GPress, a novel framework that significantly compresses genomic annotation files (GFF/GTF) and associated expression data, achieving up to a 98% size reduction. GPress enables efficient querying of this compressed data, making it a valuable tool for managing the ever-growing volume of genomic information.

What Problem Does This Paper Address?

Genomic sequencing generates vast amounts of data, which are often summarized in annotation formats like GFF, GTF, and GFF3. These files describe features of DNA, RNA, and proteins, and are crucial for biological research. As the number and size of these annotation files grow, efficient storage and rapid retrieval become critical challenges. Existing tools, while good at processing or accessing data, often fail to address the storage problem, sometimes even increasing file sizes. This leads to significant storage overhead and slower data handling, hindering large-scale genomic analyses.

Methods

GPress tackles the compression and querying problem by employing a multi-pronged approach:

  • Column-wise Transformation and Compression: GPress parses GFF files column by column, applying specific transformations to exploit correlations within and between columns. For instance, instead of storing raw start and end positions, it stores the start position and the difference between start/end positions, and differences between consecutive features. These transformed data streams are then compressed using the BSC (Block-Sorting Compressor) algorithm, known for its high performance.
  • Block-based Compression for Random Access: To enable selective querying without decompressing the entire file, GPress divides the GFF file into blocks. Each block is compressed independently. This allows GPress to only decompress the relevant blocks for a given query.
  • Index Tables for Efficient Retrieval: GPress constructs index tables that map unique identifiers and coordinate ranges to specific blocks. This enables fast identification of the necessary blocks for queries, avoiding unnecessary decompression.
  • Integration of Expression Files: GPress can also compress and link feature expression files (e.g., gene expression levels) to the compressed GFF data. It uses unique identifiers to associate expression data with genomic features, allowing for simultaneous queries across both types of files.

Datasets

The authors evaluated GPress using three GFF/GTF files from different organisms:

  • Homo sapiens (Gencode v30 annotation, 1.14 GB)
  • Mus musculus (Gencode vM21 annotation, 0.80 GB)
  • Danio rerio (GRCz11.96 annotation, 0.43 GB)

An expression file derived from the TCGA-BRCA project for Homo sapiens (7.73 GB) was also used to test the integration of expression data.

Key Insights and Ideas

  • Exploiting Data Structure for Compression: The core innovation lies in GPress's ability to understand and leverage the inherent structure and correlations within GFF files (e.g., sequential ordering, repeated values in certain columns) to achieve much higher compression ratios than general-purpose compressors like gzip.
  • Balancing Compression and Access: The block-based compression strategy is crucial. It allows for significant file size reduction while retaining the ability to perform rapid, selective queries without needing to decompress the entire dataset.
  • Unified Framework for Annotation and Expression: The ability to compress and query both GFF/GTF files and their associated expression data within a single framework is a significant advantage, simplifying complex genomic analyses.

Key Results

  • High Compression Ratios: GPress achieved an average reduction of 98% in file size for GFF/GTF files. It compressed files about twice as effectively as gzip when random access was not considered, and significantly better than gffutils.
  • Fast Query Performance: GPress can retrieve all annotations for a given identifier or a range of coordinates in a matter of seconds. For example, finding all items with a unique identifier in a human GFF file took 2.47 seconds, and finding items within a coordinate range took 4.61 seconds.
  • Efficient Expression File Handling: When linking an expression file, GPress reduced its size by over 92% while still allowing queries to be resolved within seconds.
  • Comparison to Existing Tools: While gffutils offers faster retrieval, it doubles the size of the original GFF file. GPress provides a better trade-off between compression and retrieval speed.

Limitations and Future Work

  • Block Size Optimization: The optimal block size for GPress depends on the query type. For ID searches, smaller blocks are better, while for range searches, medium-sized blocks offer a good balance. The authors suggest that a block size of 2,000 genes provides a good general trade-off.
  • Future Extensions: The authors note that the modularity and flexibility of GPress make it amenable to supporting other annotation formats in the future, potentially expanding its utility beyond GFF/GTF files.

Summary updated

Abstract

1 Abstract 1.1 Motivation Sequencing data are often summarized at different annotation levels for further analysis. The general feature format (GFF) and its descendants, the gene transfer format (GTF) and GFF3, are the most commonly used data formats for genomic annotations. These files are extensively updated, queried and shared, and hence as the number of generated GFF files increases, efficient data storage and retrieval are becoming increasingly important. Existing GFF utilities for accessing these files, like gffutils and gffread , do not focus on reducing the storage space, significantly increasing it in some cases. Hence, we propose GPress , a framework for querying GFF files in a compressed form. In addition, GPress can also incorporate and compress feature expression files, supporting simultaneous queries on both files. 1.2 Results We tested GPress on several GFF files of different organisms, and showed that it achieves on average a 98% reduction in size, while being able to retrieve all annotations for a given identifier or a range of coordinates in a few seconds. For example, on a Human GFF file, GPress can find all items with a unique identifier in 2.47 seconds and all items with coordinates within the range of 1,000 to 100,000 in 4.61 seconds. In contrast, gffutils provides faster retrieval but doubles the size of the GFF files. When additionally linking an expression file, we show that GPress can reduce the size of the expression file by more than 92%, while still retrieving the information within seconds. GPress is freely available at https://github.com/qm2/gpress .
BibTeX
@misc{meng2019gpress,
  author = {Qingxi Meng and Idoia Ochoa and Mikel Hernáez},
  title = {{“GPress: a framework for querying General Feature Format (GFF) files and feature expression files in a compressed form”}},
  year = {2019},
  doi = {10.1101/833087},
  howpublished = {bioRxiv},
}
Figure