PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database

Introduction

As genomic databases expand at an unprecedented rate—exemplified by the China National GeneBank DataBase, which manages over 17,000 TB of data—the need for efficient storage solutions has become critical. While deep learning-based lossless compressors (LLCs) have shown promise by using neural networks to predict base probabilities for entropy encoding, they face significant practical hurdles. Current LLCs often suffer from slow processing speeds, high memory overhead, and the "cold-start" problem, where models perform poorly during the initial stages of learning a new dataset. Furthermore, many models lack robustness, sometimes producing compressed files larger than the original data when applied to small datasets due to the overhead of saving model parameters.

This paper introduces PMKLC (Parallel Multi-Knowledge Learning-based Compressor), a novel framework designed to overcome these bottlenecks through a combination of multi-knowledge modeling, automated model selection, and multi-GPU acceleration.

The PMKLC Architecture

PMKLC addresses the limitations of existing LLCs through several key innovations in its Automated Multi-Knowledge Compression Framework (AMKLCF).

Multi-Knowledge Modeling

Instead of relying on a single architecture, PMKLC utilizes three distinct models to capture different data characteristics:

1. Static Public Model (SPuM): Pre-trained on diverse genomic sources, this model provides general baseline knowledge, effectively mitigating the cold-start problem.
2. Static Private Model (SPrM): This model is trained to learn the global features specific to the dataset currently being compressed.
3. Dynamic Model (DM): A Transformer-based model that updates in real-time during the compression process to capture local redundancies and shifting data distributions.

Automated Model Selector

To ensure robustness across varying data scales, PMKLC includes an automated selector. For datasets smaller than or equal to 500MB, the system prioritizes the SPuM to avoid the storage overhead of a large private model. For datasets exceeding 500MB, it activates the SPrM. A "Probability Mixer" then fuses the outputs of these active models to provide the final probability distribution for the entropy encoder.

GPU-Accelerated ($s$,$k$)-mer Encoding

Before data reaches the neural networks, PMKLC employs a sliding window approach to translate raw genomic strings (A, C, G, T) into numerical substrings. This ($s$,$k$)-mer encoding significantly reduces the sequence length the model must process, which in turn lowers the computational load and boosts overall throughput.

Parallelization and Scalability

To handle massive genomic backups, PMKLC introduces Step-wise Model Passing (SMP) for multi-GPU scaling (PMKLC-M mode). In traditional parallel setups, each GPU would suffer its own cold-start penalty as it begins processing its assigned chunk of data. SMP solves this by training the dynamic model on the first 5% of data on the primary GPU and then passing that initialization state to subsequent GPUs. This allows for massive parallelization without sacrificing the compression ratio.

Experimental Results

The authors evaluated PMKLC against 14 baselines, including seven traditional compressors (e.g., Gzip, LZMA2) and seven learning-based models (e.g., DeepZip, DNA-BiLSTM, TRACE), across 15 real-world genomic datasets.

• Compression Ratio (CR): PMKLC achieved an average CR of approximately 1.845 bits/base, representing an improvement of up to 73.6% over learning-based baselines like DNA-BiLSTM.
• Throughput: The system demonstrated significant speedups. The single-GPU mode (PMKLC-S) improved throughput by 3.03x, while the multi-GPU mode (PMKLC-M) reached a 10.71x speedup (approximately 60.89 KB/s) compared to RNN-based competitors.
• Resource Efficiency: PMKLC-M reduced CPU memory usage by up to 83.9% and GPU memory usage by 58.7% compared to the MSDLC model.
• Robustness: The automated model selector led to a 94.5% improvement in compression robustness across varying dataset sizes, ensuring that the model overhead never outweighs the data savings.

Conclusion

PMKLC represents a significant step forward in genomic data management by transforming deep learning compression from a theoretical exercise into a high-throughput, resource-efficient tool. By solving the cold-start problem and automating model selection, it provides a robust solution for large-scale genomic databases. Future work will focus on extending this parallel framework to other biological data structures, including lossy medical imaging and genomic quality scores.

@inproceedings{sun2025pmklc,
  author = {Hui Sun and Yanfeng Ding and Liping Yi and Huidong Ma and Gang Wang and Xiaoguang Liu and Cheng Zhong and Wentong Cai},
  title = {{PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
}