PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases

Qingda Hu, Xinjun Yang, Feifei Li, Junru Li, Lin Ya, Yuqi Zhou, Yicong Zhu, Junwei Zhang, Rongbiao Xie, Zhou Ling, Bin Wu, Wenchao Zhou

TL;DR

PolarStore is a high-performance, dual-layer compression system designed for large-scale cloud-native databases like Alibaba's PolarDB. By co-designing flexible software algorithms with specialized "in-storage" hardware (PolarCSD), it achieves a 3.55x compression ratio and a 60% reduction in storage costs across 100 PB of data—all while maintaining performance parity with uncompressed clusters.

What Problem Does This Paper Address?

In the world of cloud-native Relational Database Management Systems (RDBMS), storage costs are a massive pain point. While data compression is the obvious solution, existing methods force a "lose-lose" trade-off:

  1. Software-based compression is flexible but consumes heavy CPU resources and introduces complex indexing overhead. Because storage hardware typically aligns data to 4KB blocks, software compression often leaves "holes" of wasted space (fragmentation).
  2. Hardware-based compression (using Computational Storage Drives or CSDs) is fast and offloads work from the CPU, but it is "inflexible." These drives are usually hard-coded with a single algorithm and fixed 4KB input sizes, making them inefficient for the diverse, shifting workloads of a modern database.

The authors tackle the challenge of achieving high space efficiency without sacrificing the low I/O latency required for mission-critical database operations.

Methods

The core of PolarStore is a dual-layer compression architecture supported by database-specific optimizations:

  • Dual-Layer Mechanism: The software layer compresses 16KB database pages into 4KB-aligned blocks using flexible algorithms (like LZ4 or Zstd). Then, the hardware layer (PolarCSD) further compresses these blocks into byte-aligned physical addresses using a hardware-accelerated Gzip engine. This allows the system to use the drive's Flash Translation Layer (FTL) to handle fine-grained indexing without software overhead.
  • Adaptive Algorithm Selection: PolarStore doesn't just pick one algorithm. It dynamically chooses between LZ4 (fast) and Zstd (high compression) based on current CPU utilization and a "benefit vs. overhead" threshold. If Zstd can save an entire 4KB I/O block without too much latency penalty, the system switches to it.
  • Redo Log Bypass (Opt#1): Redo logs are the most latency-sensitive part of a database. PolarStore bypasses compression entirely for these logs, instead writing them to high-performance Intel Optane SSDs to ensure transaction commits remain lightning-fast.
  • Per-Page Log (Opt#3): To fix "read amplification" (where one page read requires multiple scattered log reads), PolarStore proactively co-locates redo logs with their corresponding data pages in a dedicated log space.
  • Compression-Aware Scheduling: At the cluster level, the system uses a 2D scheduling algorithm that balances both logical space (how much data the user thinks they have) and physical space (how much room is left on the flash chips) to prevent some servers from becoming "physically full" while appearing "logically empty."

Datasets

The authors evaluated PolarStore using a mix of synthetic benchmarks and real-world production data:

  • Sysbench: Used for end-to-end performance testing (OLTP Read-Write, Read-Only, etc.).
  • Production Datasets: Four distinct datasets dumped from Alibaba Cloud's production environment, covering Finance, F&B (Food & Beverage), Wiki, and Air Transport sectors. These provided a realistic variety of data entropy and access patterns.

Key Insights and Ideas

  • The "Gzip-on-LZ4" Synergy: One of the most surprising findings is that hardware Gzip can effectively further compress the output of software LZ4. This is because LZ4 lacks an entropy coding stage (like Huffman coding), which Gzip provides. This "win-win" allows for fast software processing followed by efficient hardware "cleanup."
  • FTL as an Indexer: By leveraging the SSD's internal Flash Translation Layer to manage byte-level offsets, PolarStore avoids the "4KB alignment tax" that plagues traditional file systems, achieving much higher density.
  • Stability is Hard: The transition from PolarCSD 1.0 (host-managed FTL) to 2.0 (device-managed FTL) was driven by real-world "slow I/O" events. Moving the FTL logic back into the drive's ARM controller eliminated CPU/memory contention on the host server, which is critical for "five-nines" availability.

Key Results

  • Massive Scale: PolarStore is deployed on thousands of nodes, managing over 100 PB of data.
  • Cost & Space: It achieves a 3.55x compression ratio, leading to a ~60% reduction in storage costs.
  • Performance Parity: Despite the overhead of compression, the "C2" cluster (PolarCSD 2.0 + Dual-layer) matches the throughput and latency of "N2" clusters using standard high-performance NVMe SSDs (Intel P5510).
  • Tail Latency: The per-page log optimization reduced P95 latency by 28.9% to 39.5% in scenarios with heavy log pressure.

Limitations and Future Work

  • Table-Level Context: The current system compresses pages independently. The authors suggest that using table-level shared dictionaries could further improve compression ratios by exploiting similarities across different pages in the same table.
  • Redo Log Redundancy: While redo logs currently bypass compression for speed, the authors note that Erasure Coding (EC) for logs is a potential future direction to reduce the cost of the current 3-way replication model.
  • Deduplication: The paper notes that page-level deduplication is currently rare in RDBMS and was not a focus, though it remains a theoretical area for further space savings.

Summary updated

Abstract

In recent years, resource elasticity and cost optimization have become essential for RDBMSs. While cloud-native RDBMSs provide elastic computing resources via disaggregated computing and storage, storage costs remain a critical user concern. Consequently, data compression emerges as an effective strategy to reduce storage costs. However, existing compression approaches in RDBMSs present a stark trade-off: software-based approaches incur significant performance overheads, while hardware-based alternatives lack the flexibility required for diverse database workloads. In this paper, we present PolarStore, a compressed shared storage system for cloud-native RDBMSs. PolarStore employs a dual-layer compression mechanism that combines in-storage compression in PolarCSD hardware with lightweight compression in software. This design leverages the strengths of both approaches. PolarStore also incorporates database-oriented optimizations to maintain high performance on critical I/O paths. Drawing from large-scale deployment experiences, we also introduce hardware improvements for PolarCSD to ensure host-level stability and propose a compression-aware scheduling scheme to improve cluster-level space efficiency. PolarStore is currently deployed on thousands of storage servers within PolarDB, managing over 100 PB of data. It achieves a compression ratio of 3.55 and reduces storage costs by approximately 60%. Remarkably, these savings are achieved while maintaining performance comparable to uncompressed clusters.
BibTeX
@misc{hu2025polarstore,
  author = {Qingda Hu and Xinjun Yang and Feifei Li and Junru Li and Lin Ya and Yuqi Zhou and Yicong Zhu and Junwei Zhang and Rongbiao Xie and Zhou Ling and Bin Wu and Wenchao Zhou},
  title = {{PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases}},
  year = {2025},
  eprint = {2511.19949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.19949},
}
Figure