An introduction to MPEG-G, the new ISO standard for genomic information representation

C. Albert, Tom Paridaens, Jan Voges, Daniel Naro, Junaid J. Ahmad, M. Ravasi, Daniele Renzi, Giorgio Zoia, Paolo Ribeca, Idoia Ochoa, Marco Mattavelli, Jaime Delgado, Mikel HernáezbioRxiv 2018 · DOI

TL;DR

The MPEG-G standard is a new ISO standard designed to address the massive data explosion from high-throughput sequencing. It provides a comprehensive framework for compressing, storing, transporting, and processing genomic data, offering significant improvements over existing formats in terms of efficiency, functionality, and interoperability.

What Problem Does This Paper Address?

The rapid advancement of High-Throughput Sequencing (HTS) technologies has led to an exponential increase in genomic data generation. This deluge of data, projected to surpass astronomical data volumes, presents significant challenges for storage, transmission, and processing. Current formats like FASTQ, SAM, and BAM, while functional, lack efficient compression, robust data protection, and seamless integration capabilities, hindering the widespread use of genomic information. MPEG-G aims to solve this by providing a standardized, efficient, and feature-rich solution for genomic data management.

Methods

MPEG-G is built upon a rigorous standardization process involving a Call for Proposals and evaluation of submitted technologies. The standard defines a normative decoding process, allowing for continuous innovation in encoder implementations. Key methodological aspects include:

  • Genomic Records: A fundamental data structure that groups information related to a single DNA template, including raw sequencing data, alignment information, identifiers, and quality scores.
  • Data Classes: Genomic Records are classified into six categories (P, N, M, I, HM, U) based on their alignment characteristics to reference genomes, enabling efficient data organization and selective access.
  • Descriptor Streams: Information within clustered Genomic Records is further split into Descriptor Streams, allowing for tailored encoding parameters based on the statistical properties of each descriptor type.
  • Access Units: The fundamental unit for coded genomic information that can be independently accessed. They contain headers with metadata and data Blocks of compressed genomic data.
  • MPEG-G File Format: A hierarchical digital container for storing compressed genomic data, organized into Dataset Groups, Datasets, and Access Units, facilitating efficient querying and selective access.
  • Transport Format: Structures for packetized data transport over networks, enabling streaming and on-the-fly indexing.
  • Compression Strategies: MPEG-G supports various compression modes for raw sequencing data (high compression/indexing vs. low latency), aligned reads (reference-based vs. reference-free), quality values (lossless vs. quantized), and read identifiers.
  • Entropy Coding: Utilizes Context-Adaptive Binary Arithmetic Coding (CABAC) for efficient compression.
  • Normative Decoding Process: A precisely defined process for extracting information from MPEG-G bitstreams, ensuring interoperability.
  • Metadata and APIs: Support for integrating metadata, access control, integrity verification, and authentication.
  • Reference Software and Conformance Testing: Tools and procedures to aid implementation and validate compliance.

Datasets

The paper mentions the MPEG-G Genomic Information Database, which is a collection of statistically meaningful sequencing data used for assessing compression performance. This database includes data from various experiment types (WGS, RNA-seq, metagenomics), sequenced organisms (H. sapiens, D. melanogaster, E. coli, etc.), and sequencing technologies (Illumina, PacBio, Nanopore, Ion Torrent). The database is publicly accessible via GitHub.

Key Insights and Ideas

  • Beyond Simple Compression: MPEG-G is not just about making files smaller; it's a comprehensive system for managing genomic data, integrating compression with crucial functionalities like selective access, data protection, and streaming.
  • Interoperability is Key: By defining a normative decoding process and supporting conversion from existing formats (FASTQ, SAM, BAM), MPEG-G aims to seamlessly integrate into existing bioinformatics pipelines.
  • Flexibility for Diverse Needs: The standard offers different compression modes tailored for various applications, from long-term archival requiring high compression to real-time streaming demanding low latency.
  • Data Protection as a First-Class Citizen: MPEG-G incorporates robust mechanisms for privacy enforcement, encryption, and integrity verification, addressing critical concerns in sensitive genomic data handling.
  • A Foundation for Future Innovation: By standardizing the decoding process, MPEG-G allows for continuous improvement and diversification of encoding technologies, ensuring the standard remains relevant and performant over time.

Key Results

While the paper focuses on introducing the standard rather than presenting specific experimental results, it highlights the potential for significant compression gains:

  • Raw Sequencing Data: MPEG-G encoders can achieve over 85% reduction for raw FASTQ files (lossless), compared to over 70% with gzip.
  • Aligned Data: Compression comparable to or better than CRAM and BAM is achievable for aligned data, with examples showing a 437 GB SAM file compressed to about 63 GB using MPEG-G technologies.
  • Quality Values: Quantization of quality values can lead to overall compression gains of 10x over BAM, potentially improving variant calling performance.
  • Entropy Coding: Achieved compression down to about 21% of the uncompressed size at an average processing speed of over 25 MiB/s per descriptor stream.

The paper emphasizes that these are indicative baseline examples, and actual performance will vary based on encoder implementations.

Limitations and Future Work

  • Encoder Independence: The standard only specifies the decoding process, meaning encoding performance (speed and compression ratio) is dependent on specific encoder implementations and is expected to improve over time.
  • Overhead: Some minor overhead is expected due to the encapsulation of data within the MPEG-G file format, which provides additional functionality beyond pure compression.
  • Future Work: The paper implies that the ongoing development and maintenance of the standard by ISO/IEC will ensure its perenniality and encourage further investment and innovation in MPEG-G technologies. The standard is designed to evolve and improve, fostering a genomic ecosystem similar to that of digital media.

Summary updated

Abstract

Abstract The MPEG-G standardization initiative is a coordinated international effort to specify a compressed data format that enables large scale genomic data to be processed, transported and shared. The standard consists of a set of specifications (i.e., a book) describing: i) a nor-mative format syntax, and ii) a normative decoding process to retrieve the information coded in a compliant file or bitstream. Such decoding process enables the use of leading-edge com-pression technologies that have exhibited significant compression gains over currently used formats for storage of unaligned and aligned sequencing reads. Additionally, the standard provides a wealth of much needed functionality, such as selective access, data aggregation, ap-plication programming interfaces to the compressed data, standard interfaces to support data protection mechanisms, support for streaming and a procedure to assess the conformance of implementations. ISO/IEC is engaged in supporting the maintenance and availability of the standard specification, which guarantees the perenniality of applications using MPEG-G. Fi-nally, the standard ensures interoperability and integration with existing genomic information processing pipelines by providing support for conversion from the FASTQ/SAM/BAM file formats. In this paper we provide an overview of the MPEG-G specification, with particular focus on the main advantages and novel functionality it offers. As the standard only specifies the decoding process, encoding performance, both in terms of speed and compression ratio, can vary depending on specific encoder implementations, and will likely improve during the lifetime of MPEG-G. Hence, the performance statistics provided here are only indicative baseline examples of the technologies included in the standard.
BibTeX
@misc{albert2018introduction,
  author = {C. Albert and Tom Paridaens and Jan Voges and Daniel Naro and Junaid J. Ahmad and M. Ravasi and Daniele Renzi and Giorgio Zoia and Paolo Ribeca and Idoia Ochoa and Marco Mattavelli and Jaime Delgado and Mikel Hernáez},
  title = {{An introduction to MPEG-G, the new ISO standard for genomic information representation}},
  year = {2018},
  doi = {10.1101/426353},
  howpublished = {bioRxiv},
}
Figure