TL;DR
This paper introduces GPress, a novel framework that significantly compresses genomic annotation files (GFF/GTF) and associated expression data, achieving up to a 98% size reduction. GPress enables efficient querying of this compressed data, making it a valuable tool for managing the ever-growing volume of genomic information.
What Problem Does This Paper Address?
Genomic sequencing generates vast amounts of data, which are often summarized in annotation formats like GFF, GTF, and GFF3. These files describe features of DNA, RNA, and proteins, and are crucial for biological research. As the number and size of these annotation files grow, efficient storage and rapid retrieval become critical challenges. Existing tools, while good at processing or accessing data, often fail to address the storage problem, sometimes even increasing file sizes. This leads to significant storage overhead and slower data handling, hindering large-scale genomic analyses.
Methods
GPress tackles the compression and querying problem by employing a multi-pronged approach:
- Column-wise Transformation and Compression: GPress parses GFF files column by column, applying specific transformations to exploit correlations within and between columns. For instance, instead of storing raw start and end positions, it stores the start position and the difference between start/end positions, and differences between consecutive features. These transformed data streams are then compressed using the BSC (Block-Sorting Compressor) algorithm, known for its high performance.
- Block-based Compression for Random Access: To enable selective querying without decompressing the entire file, GPress divides the GFF file into blocks. Each block is compressed independently. This allows GPress to only decompress the relevant blocks for a given query.
- Index Tables for Efficient Retrieval: GPress constructs index tables that map unique identifiers and coordinate ranges to specific blocks. This enables fast identification of the necessary blocks for queries, avoiding unnecessary decompression.
- Integration of Expression Files: GPress can also compress and link feature expression files (e.g., gene expression levels) to the compressed GFF data. It uses unique identifiers to associate expression data with genomic features, allowing for simultaneous queries across both types of files.
Datasets
The authors evaluated GPress using three GFF/GTF files from different organisms:
- Homo sapiens (Gencode v30 annotation, 1.14 GB)
- Mus musculus (Gencode vM21 annotation, 0.80 GB)
- Danio rerio (GRCz11.96 annotation, 0.43 GB)
An expression file derived from the TCGA-BRCA project for Homo sapiens (7.73 GB) was also used to test the integration of expression data.
Key Insights and Ideas
- Exploiting Data Structure for Compression: The core innovation lies in GPress's ability to understand and leverage the inherent structure and correlations within GFF files (e.g., sequential ordering, repeated values in certain columns) to achieve much higher compression ratios than general-purpose compressors like gzip.
- Balancing Compression and Access: The block-based compression strategy is crucial. It allows for significant file size reduction while retaining the ability to perform rapid, selective queries without needing to decompress the entire dataset.
- Unified Framework for Annotation and Expression: The ability to compress and query both GFF/GTF files and their associated expression data within a single framework is a significant advantage, simplifying complex genomic analyses.
Key Results
- High Compression Ratios: GPress achieved an average reduction of 98% in file size for GFF/GTF files. It compressed files about twice as effectively as gzip when random access was not considered, and significantly better than gffutils.
- Fast Query Performance: GPress can retrieve all annotations for a given identifier or a range of coordinates in a matter of seconds. For example, finding all items with a unique identifier in a human GFF file took 2.47 seconds, and finding items within a coordinate range took 4.61 seconds.
- Efficient Expression File Handling: When linking an expression file, GPress reduced its size by over 92% while still allowing queries to be resolved within seconds.
- Comparison to Existing Tools: While gffutils offers faster retrieval, it doubles the size of the original GFF file. GPress provides a better trade-off between compression and retrieval speed.
Limitations and Future Work
- Block Size Optimization: The optimal block size for GPress depends on the query type. For ID searches, smaller blocks are better, while for range searches, medium-sized blocks offer a good balance. The authors suggest that a block size of 2,000 genes provides a good general trade-off.
- Future Extensions: The authors note that the modularity and flexibility of GPress make it amenable to supporting other annotation formats in the future, potentially expanding its utility beyond GFF/GTF files.
Summary updated