DupHunter: Flexible High-Performance Deduplication for Docker Registries

Nannan Zhao, Hadeel Albahar, Subil Abraham, Keren Chen, Vasily Tarasov, Dimitrios Skourtis, Lukas Rupprecht, Ali Anwar, Ali Raza ButtUSENIX ATC 2020 · OpenReview

TL;DR

Docker registries are struggling with massive storage demands due to redundant container images. DupHunter introduces a novel registry architecture that natively deduplicates layers, significantly reducing storage space while also optimizing layer retrieval performance through intelligent caching and parallel reconstruction.

What Problem Does This Paper Address?

The explosion of containerized applications has led to an exponential growth in container images stored in registries like Docker Hub. These images, structured in layers, often contain a high degree of redundancy, leading to substantial storage and network overhead. Existing deduplication techniques, while effective at reducing raw data size, introduce significant latency during layer retrieval (pulling images), which directly impacts container startup times. This paper addresses the critical need for a Docker registry solution that can achieve high storage efficiency through deduplication without sacrificing performance, particularly for layer retrieval.

Methods

DupHunter proposes a new Docker registry architecture with several key innovations:

  • Native Layer Deduplication: Unlike traditional approaches that might deduplicate at the block level after decompression, DupHunter deduplicates at the file level within layers.
  • Flexible Deduplication Modes: DupHunter offers configurable modes (B-mode n and S-mode) that allow users to balance storage savings with performance and durability. This includes keeping a certain number of full layer replicas for performance while deduplicating others.
  • Two-Tier Storage Hierarchy: The architecture uses a primary cluster (P-servers) for fast access to intact layer replicas and a deduplication cluster (D-servers) for storing deduplicated data.
  • Intelligent Caching:
    • Prefetch Cache (P-servers): An in-memory cache that predicts and prefetches layers likely to be accessed soon based on image manifests and user access patterns.
  • Preconstruct Cache (D-servers): An on-disk cache that stores pre-reconstructed layers, further reducing retrieval latency.
  • Parallel Layer Reconstruction: DupHunter divides layers into slices and distributes them across D-servers, enabling parallel reconstruction of layers when needed.
  • User Access Pattern Analysis: The system leverages historical data to predict user behavior, distinguishing between "always-pull" and "pull-once" users to optimize caching strategies.

Datasets

The evaluation uses real-world workloads from the IBM Cloud Registry traces, spanning approximately 80 days across four production clusters (Dal, Fra, Lon, and Syd). This dataset consists of popular Docker images with a significant number of compressed layers. The authors also mention using a dataset of 0.93 TB of popular Docker images, totaling 2 TB after decompression.

Key Insights and Ideas

  • File-level deduplication within layers is crucial: Simply deduplicating compressed tarballs is ineffective. Decompressing and then deduplicating files within layers is necessary to expose redundancy.
  • The performance penalty of naive deduplication is severe: Existing deduplication techniques can increase layer pull latencies by up to 98x, making them impractical for registries where fast container startup is essential.
  • User access patterns are predictable and exploitable: By analyzing how users interact with layers, DupHunter can intelligently cache and prefetch data, significantly mitigating retrieval overhead.
  • A tiered storage approach is effective: Separating intact replicas from deduplicated data allows for a balance between performance and storage efficiency.
  • Flexible trade-offs are key: Not all users or workloads require the same level of deduplication. Offering configurable modes allows for tailored solutions.

Key Results

  • Storage Reduction: In the highest deduplication mode (GB-EC), DupHunter reduces storage space by up to 6.9x compared to current implementations.
  • Performance Improvement: In the highest performance mode (B-mode 3), DupHunter can reduce GET layer latency by up to 2.8x compared to the state-of-the-art.
  • Cache Effectiveness: The prefetch cache (ARC+P-UB) achieved hit ratios of up to 0.96 with a small cache size (15%), outperforming other algorithms by up to 4.2x.
  • Layer Restore Overhead Reduction: DupHunter significantly decreases layer restoring overhead compared to naive approaches. For example, DupHunter B-mode 0 with VDO (GF+LB-R) has a GET layer latency only 2.87x slower than the baseline, whereas a VDO-only scheme was 60x slower.
  • Scalability: The preconstruct cache shows good scalability, with average wait times remaining low even with an increased number of concurrent clients.

Limitations and Future Work

  • Block-level Deduplication Integration: While DupHunter focuses on file-level deduplication, the authors suggest integrating with block-level deduplication (e.g., VDO, Ceph) for even higher deduplication ratios. However, they note this can increase layer restoring overhead.
  • Erasure Coding: The paper explores integrating with erasure coding (GB-EC) for improved space efficiency, but this also comes with higher restoring overhead.
  • Chunk Loss Impact: The authors acknowledge that deduplication can increase the impact of chunk loss on fault tolerance and maintain at least three copies of a layer to mitigate this.
  • Future Work: The paper suggests further exploration of block-level deduplication and erasure coding integration, as well as optimizing placement strategies for deduplicated data.

Summary updated

BibTeX
@inproceedings{zhao2020duphunter,
  author = {Nannan Zhao and Hadeel Albahar and Subil Abraham and Keren Chen and Vasily Tarasov and Dimitrios Skourtis and Lukas Rupprecht and Ali Anwar and Ali Raza Butt},
  title = {{DupHunter: Flexible High-Performance Deduplication for Docker Registries}},
  booktitle = {USENIX ATC},
  year = {2020},
  url = {https://openreview.net/forum?id=CPnAHqjVZ2},
}
Figure