AegonKV: A High Bandwidth, Low Tail Latency, and Low Storage Cost KV-Separated LSM Store with SmartSSD-based GC Offloading

Zhuohui Duan, Hao Feng, Haikun Liu, Xiaofei Liao, Hai Jin, Bangyu LiFAST 2025 · OpenReview

TL;DR

This paper introduces AegonKV, a novel KV-separated LSM store that offloads garbage collection (GC) to SmartSSD hardware. This approach significantly boosts throughput, slashes tail latency, and reduces storage costs by overcoming the limitations of traditional GC implementations.

What Problem Does This Paper Address?

Key-value (KV) stores, particularly those based on Log-Structured Merge (LSM) trees, are crucial for data-intensive applications due to their excellent write performance. However, traditional LSM trees suffer from high read and write overheads during compaction. KV separation addresses this by decoupling keys and values, reducing LSM tree size and compaction overhead.

Despite these benefits, KV separation introduces new challenges, primarily in managing the "Value region," especially during garbage collection (GC). GC is essential for reclaiming space occupied by expired data. Existing GC optimizations for KV-separated systems often involve trade-offs between CPU and I/O overheads, failing to simultaneously achieve high throughput, low tail latency, and efficient space usage. This limitation hinders their real-world applicability. The authors identify that existing GC strategies, whether "Direct GC" or "Compaction-triggered GC," face bottlenecks due to host CPU and I/O contention, leading to performance degradation.

Methods

AegonKV tackles these challenges by proposing a "three-birds-one-stone" solution that leverages SmartSSD's near-data processing capabilities. The core innovation is SmartSSD-based GC offloading.

Here's a breakdown of their approach:

  • Asynchronous GC Offloading: AegonKV offloads GC operations to the FPGA within the SmartSSD. This allows GC to run in parallel with host-side LSM read/write operations, preventing competition for bandwidth and CPU resources.
  • Offload-Friendly Data Structures:
    • ValidMap: A compact bitmap data structure is introduced to efficiently track the validity of data within Blob files. This is generated from compaction results and helps in granular GC targeting.
  • GC Manager: This component orchestrates the preparation of ValidMaps and manages GC jobs.
  • GC Scheduler: This module on the SmartSSD efficiently drives GC operations using specialized units:
  • I/O Control Unit: Leverages SmartSSD's P2P (peer-to-peer) data path for efficient data transfer between SSD and DRAM.
  • Compute Unit Control Unit: Manages the execution of GC tasks on the FPGA.
  • Metadata Install Unit: Handles the installation of GC metadata (new KV addresses) back into the LSM structure.
  • GC Friendly Metadata Install: To avoid the performance penalty of validating every metadata write, AegonKV implements a deferred validation approach. Metadata is batched and written to the MemTable without immediate validation. Consistency is ensured by checking during Get and compaction operations, where the system determines the latest version by comparing data addresses. This strategy minimizes the overhead of metadata updates.
  • Hardware-Optimized Compute Units (CUs): The FPGA is programmed with highly parallel CUs to perform GC stages like input decoding, data filtering, fetching, and output encoding, minimizing host intervention.

Datasets

The authors evaluated AegonKV using:

  • YCSB Benchmark: A standard benchmark for evaluating database performance, with various workloads (A-F) representing different read/write/scan ratios.
  • Realistic Production Workloads:
    • Social Graph: A workload simulating social graph operations.
  • Twitter Cache Cluster Traces: Three traces (cluster 39, 19, and 51) representing write-intensive, mixed, and read-intensive cache workloads, replaying the first 500 million operations.

Key Insights and Ideas

  • GC Offloading is Key: The primary bottleneck in KV-separated LSM stores is GC, not compaction itself. Offloading GC to SmartSSD is a game-changer, effectively decoupling it from host resources.
  • SmartSSD's Near-Data Processing is Crucial: The ability of SmartSSD to perform computations directly on the storage device, bypassing the PCIe bus and host CPU, is fundamental to AegonKV's success.
  • Deferred Validation for Metadata: Optimizing metadata writes by deferring validation significantly reduces overhead and improves performance without compromising data consistency.
  • Trade-offs are Inherent, but Solvable: Previous KV-separated systems demonstrated trade-offs. AegonKV shows that by intelligently offloading GC and using optimized data structures, these trade-offs can be largely overcome.
  • Hardware-Software Co-design: The success of AegonKV relies on the synergistic design of hardware (SmartSSD FPGA) and software components (GC Manager, Scheduler).

Key Results

AegonKV demonstrated significant improvements across multiple metrics compared to state-of-the-art KV-separated systems (Titan, DiffKV, BlobDB) and traditional LSM stores (RocksDB):

  • Throughput: Achieved the largest throughput improvement, ranging from 1.28x to 3.3x in YCSB workloads and up to 2.16x in production workloads. In write-intensive scenarios, it closely approached the performance of "Titan w/o GC."
  • Tail Latency (99%): Showed a significant reduction of 37%-66%. In write-intensive workloads, AegonKV achieved the lowest tail latency, comparable to the ideal "Titan w/o GC" scenario.
  • Space Usage: Reduced space overhead by 15%-85%. In write-intensive workloads, AegonKV's space usage was comparable to RocksDB, while DiffKV and BlobDB showed much higher space amplification.
  • Compaction I/O: Significantly reduced compaction I/O, especially compared to Compaction-triggered GC systems like DiffKV and BlobDB.
  • Write Stall: Achieved 0 write stall in write-intensive workloads, similar to Titan but outperforming RocksDB, BlobDB, and DiffKV.
  • CPU Usage: AegonKV achieved approximately 20% CPU savings compared to other systems, attributed to offloading GC computation.
  • Energy Efficiency: AegonKV achieved the highest energy efficiency, improving by 15.9% to 69.8%.

Limitations and Future Work

  • Hardware Dependency: AegonKV's performance benefits are tied to the availability and capabilities of SmartSSD-like computational storage devices.
  • FPGA Resource Limitations: While optimized, the FPGA within SmartSSD has finite resources, requiring careful management and potentially limiting the scale of offloading.
  • Read-Intensive Performance: In purely read-intensive scenarios, AegonKV's performance was slightly lower than Titan w/o GC due to the overhead of maintaining ValidMaps. However, the difference was marginal.
  • Future Work: The authors suggest exploring further optimizations for GC offloading, potentially by rescheduling data movement between hardware and host to further reduce channel congestion. They also highlight the potential for SmartSSDs in other KV-separated system optimizations.

Summary updated

BibTeX
@inproceedings{duan2025aegonkv,
  author = {Zhuohui Duan and Hao Feng and Haikun Liu and Xiaofei Liao and Hai Jin and Bangyu Li},
  title = {{AegonKV: A High Bandwidth, Low Tail Latency, and Low Storage Cost KV-Separated LSM Store with SmartSSD-based GC Offloading}},
  booktitle = {FAST},
  year = {2025},
  url = {https://openreview.net/forum?id=CzLuLSBU6o},
}
Figure