Compressing Genomic Data and Quality Scores

This direction is merged into another direction: genomic data scalability. Its last version stays readable and citable here.

Version 1 of 149 papers56 conceptsfrozen 2026-10-04

The cost of sequencing a human genome has plummeted from millions of dollars to just a few hundred. While this is a triumph for personalized medicine, it has created a massive infrastructure crisis. Genomic data volume is doubling every 12 to 18 months, a pace that Jan Voges and others note is outpacing the growth of physical storage capacity and may soon exceed the storage requirements of global video data.

This "data deluge," first identified during the Human Genome Project by researchers like Serafim Batzoglou and Eric Lander, has shifted the bottleneck of biology from the wet lab to the data center. To keep genomics sustainable, researchers are developing specialized compression techniques that exploit the unique mathematical properties of biological information.

The Entropy Problem of Quality Scores

A standard sequencing file (FASTQ) contains two main types of data: the nucleotide sequences (A, C, G, T) and the "quality scores" that represent the sequencer's confidence in each base call.

Counter-intuitively, the quality scores are the primary storage bottleneck. As Yongpeng Zhang and Linsen Li point out, these scores often account for 60% to 80% of a compressed file's size. The reason is entropy: while DNA sequences are highly repetitive and predictable, quality scores are noisy, use a larger alphabet, and lack inherent patterns.

To solve this, researchers like Yi Niu and Mingming Ma have developed algorithms like ACO, which rethink data traversal. Instead of standard line-by-line scanning, ACO uses a "snake-like" adaptive order to find hidden correlations between scores and the underlying DNA bases, squeezing out more redundancy than general-purpose tools like Gzip.

Lossy Compression as a Denoiser

Because quality scores are so difficult to compress losslessly, a major research direction applies Rate Distortion Theory to genomic data. The intuition is simple: do we really need to know if a base call is 99.99% certain versus 99.98% certain?

Himanshu Asnani, Idoia Ochoa, and Tsachy Weissman developed QualComp, which treats quality scores as noisy signals. By allowing for controlled "lossy" compression, they can reduce storage requirements to less than one bit per value.

Remarkably, Ochoa and Mikel Hernaez discovered that lossy compression can actually act as a "denoiser." By smoothing out insignificant fluctuations in quality scores, these methods can sometimes improve the accuracy of downstream variant calling compared to using the raw, uncompressed data.

Exploiting Biological Context

Genomic data is rarely random; it usually consists of many overlapping fragments of the same underlying sequence. Compression algorithms leverage this in two ways:

  1. Reference-based Compression: Tools like FastqZip map reads to a known reference genome and store only the differences (mutations). This is highly efficient but requires a high-quality reference.
  2. Reference-free Assembly: For novel organisms or metagenomic "soups," researchers use "light assembly." Antonio A. Ginart and Manolis Kellis introduced Assembltrie, which organizes reads into a compact trie structure, achieving near-theoretical limits of efficiency without needing a reference.

For large-scale repositories, Gunnar Ratsch and André Kahles have pioneered graph-based sparsification. Their tool, RowDiff, exploits the fact that adjacent nodes in a de Bruijn graph likely share the same biological labels. By storing only the XOR deltas (differences) between neighbors, they can reduce the size of massive annotation matrices by up to 26-fold.

The Move Toward Learned and Standardized Models

The latest frontier in this field involves deep learning and international standards. Zhanbei Cui and colleagues developed GeneFormer, adapting Transformer-XL architectures to model the long-range context of DNA. Meanwhile, Hui Sun and Liping Yi have pushed the boundaries of throughput with PMKLC, using multi-GPU acceleration to make neural compression fast enough for production environments.

Finally, to prevent a fragmented landscape of incompatible tools, Jan Voges introduced Genie, the first open-source encoder for the MPEG-G (ISO/IEC 23092) standard. This represents a shift toward treating genomic data not just as a biological artifact, but as a standardized digital medium that requires the same level of interoperability as video or audio.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compressing_genomic_data_and_quality_sco_869f63e3,
  title = {Compressing Genomic Data and Quality Scores},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compressing-genomic-data-and-quality-scores/rdr_869f63e390f644d18d29f16d87a57291},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compressing_genomic_data_and_quality_sco_869f63e3_v1,
  title = {Compressing Genomic Data and Quality Scores},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compressing-genomic-data-and-quality-scores/rdr_869f63e390f644d18d29f16d87a57291/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}