Abstract
Mass spectrometry (MS) is a fundamental tool in proteomics and metabolomics, generating vast quantities of data that challenge existing storage infrastructures. Current storage formats often rely on inefficient text-based encoding, while existing specialized compressors frequently sacrifice data precision. We present MassComp, a domain-specific, lossless compression tool designed to handle the unique numerical properties of MS data. By employing specialized algorithms for mass-to-charge (m/z) and intensity values, MassComp achieves significant size reductions—averaging 59% for entire files—while maintaining 100% data integrity.
Introduction
The rapid advancement of high-throughput mass spectrometry has led to an explosion in data volume. Large-scale repositories like MassIVE now manage over 120 terabytes of data, much of which is stored in XML-based formats such as mzXML or mzML. These formats typically encode numerical data in Base64 strings, a method that is highly inefficient for storage and transmission. While general-purpose compressors like gzip provide some relief, they lack the domain awareness necessary to exploit the specific patterns found in MS data. MassComp addresses this by providing a high-performance, lossless alternative that outperforms both general-purpose and existing specialized numerical compressors.
The Challenge of Mass Spectrometry Data Storage
The primary bottleneck in MS data storage lies in the representation of (m/z)-intensity pairs. In standard mzXML files, these pairs are stored as long strings of Base64-encoded characters within scan elements.
As shown in the figure above, a typical file consists of general metadata followed by individual scan information. The numerical data within these scans does not compress well with standard dictionary-based algorithms because the bit-level patterns of floating-point numbers appear random to general-purpose tools. Furthermore, while lossy compressors like MS-Numpress can achieve high compression ratios, they introduce precision errors that may be unacceptable for certain downstream analyses, particularly when dealing with double-precision (64-bit) data.
The MassComp Methodology: Specialized Compression
MassComp utilizes a "divide and conquer" strategy, separating the XML metadata from the numerical peak data. The metadata is compressed using standard gzip, while the (m/z)-intensity pairs are processed through two distinct, specialized algorithms that exploit the physical properties of the data.
m/z Compression
Mass-to-charge (m/z) values within a single scan are typically smooth and monotonically increasing. MassComp leverages this by converting IEEE 754 floating-point values into their hexadecimal representations and applying a variation of delta encoding. By calculating the difference between adjacent values in the hexadecimal domain, the algorithm generates strings with many leading zeros, which are then efficiently compressed using an arithmetic encoder.
Intensity Compression
Intensity values are more stochastic than m/z values, requiring a different approach. MassComp employs a "match" compression algorithm that focuses on the internal structure of the floating-point representation.
As illustrated in the IEEE 754 standard, floating-point numbers are composed of sign, exponent, and fraction bits. MassComp searches for bit-pattern matches—specifically within the sign and exponent bits—among the 15 most recently compressed values. If a match is found, a pointer is stored; otherwise, the raw data is kept. This strategy is adaptive, with the tool inspecting the first 50 values of each scan to determine the most effective matching parameters for that specific data block.
Performance Evaluation and Results
The effectiveness of MassComp was evaluated using diverse datasets from the MassIVE repository, covering both 32-bit and 64-bit precision in centroid and profile modes. The results demonstrate that MassComp is highly effective at reducing data footprints:
- Numerical Data Reduction: Achieved an average 46% reduction in size, with peak performance reaching 89% in specific cases.
- Comparative Performance: MassComp outperformed
gzipby an average of 27% and the state-of-the-art numerical compressorFPCby 40%. - Whole File Compression: When considering the entire mzXML file (metadata plus numerical data), MassComp achieved an average size reduction of 59%.
Crucially, MassComp maintains bit-for-bit accuracy, making it particularly superior for double-precision datasets where lossy compression is not an option.
Discussion and Future Directions
While MassComp offers superior compression ratios, there are areas for future optimization. Currently, decompression is slower than gzip due to the computational overhead of reassembling the XML structure and the arithmetic decoding process. Additionally, the current single-threaded implementation does not yet take advantage of the highly parallelizable nature of MS scans.
Future work will focus on implementing multi-threaded processing to improve throughput and developing an indexing system to allow researchers to access specific scans without decompressing the entire file. Furthermore, the authors plan to extend native support to the mzML format to broaden the tool's utility within the research community. In conclusion, MassComp demonstrates that by exploiting the underlying physical constraints and bit-level structure of mass spectrometry data, it is possible to achieve high compression ratios without sacrificing scientific precision.
Summary updated