TL;DR
This paper introduces SimEnc, a novel approach for deduplicating encrypted Docker images that significantly improves storage efficiency and reduces pull latency. By leveraging semantic hashing and a new "partially decoded space," SimEnc overcomes the limitations of existing methods that struggle with minor data modifications and incur high overheads.
What Problem Does This Paper Address?
Docker registries are increasingly storing encrypted images for privacy. To manage the massive growth in image data, deduplication is crucial for saving storage space. However, deduplicating encrypted data is inherently difficult because encryption aims to make data look random, while deduplication relies on identical content. Existing solutions often decompress images before encryption and deduplication, which leads to two major problems: (1) even minor changes in image content can break deduplication, and (2) decompressing and recompressing layers significantly increases storage requirements and user pull latency. This paper tackles the challenge of achieving high-performance deduplication for encrypted Docker images without sacrificing security or introducing excessive overhead.
Methods
SimEnc introduces several key innovations:
- Partially Decoded Space: Instead of fully decompressing Docker image layers, SimEnc explores a "partially decoded space" that involves only Huffman decoding. This space offers a better trade-off between deduplication ratio and latency compared to the fully decompressed space.
- Fast Similarity Space Selection: A mechanism that leverages Huffman trees (found in layer headers) to quickly determine the most suitable space (compressed or partially decoded) for deduplication. This allows for efficient layer processing.
- Semantic-aware MLE: The core of SimEnc's approach. It integrates semantic hashing into Message-Locked Encryption (MLE).
- Chunk Semantic Extraction: Uses a trained hash network to extract semantic information from Docker layers, overcoming the challenge of diverse file types within images. It employs a novel pseudo-labeling strategy based on byte-level similarity and compression ratios.
- Similarity-Preserving Key Generation: Addresses the critical need for identical keys for identical chunks. It uses DBSCAN clustering on semantic hashes to group similar chunks and assign them a common key, ensuring that similar data can be deduplicated after encryption.
- Flexible Deduplication Modes: SimEnc offers three modes (Basic, High, and Flexible) to balance deduplication ratio, latency, and throughput based on user needs.
Datasets
The paper evaluates SimEnc using a variety of datasets and workloads:
- Ubuntu [24] and Couchbase [20] images: Used for initial experiments and analysis of similarity spaces.
- IBM trace datasets [5, 35]: Four production registry workloads (Dal, Fra, Lon, Syd) and a larger random IBM dataset were used to evaluate performance under realistic conditions. These datasets consist of sequential image versions, which are particularly challenging for deduplication due to incremental modifications.
Key Insights and Ideas
- Partially decoded space is surprisingly effective: The paper demonstrates that a partially decoded space (using only Huffman decoding) can yield better deduplication ratios than fully decompressed spaces, challenging conventional wisdom.
- Semantic similarity is key for encrypted deduplication: Encryption destroys exact byte-level similarity, but semantic similarity can be preserved and leveraged for effective deduplication.
- Minor changes are a major hurdle for traditional MLE: Existing MLE techniques are highly sensitive to small data modifications, significantly reducing deduplication effectiveness. SimEnc's semantic approach mitigates this.
- Balancing trade-offs is essential: SimEnc's flexible modes highlight that there's no one-size-fits-all solution, and different applications require different balances between storage savings, latency, and throughput.
Key Results
- Deduplication Ratio: SimEnc achieves significantly higher deduplication ratios compared to state-of-the-art methods. In the High Deduplication mode, it outperforms the enhanced AWS Lambda by up to 54.2% and the basic AWS Lambda by up to 261.7% on the Ubuntu dataset.
- Pull Latency: SimEnc not only improves storage but also reduces pull latency. In the IBM (random) workload, SimEnc's average latency is 58.4% lower than DupHunter. Even with re-encoding, SimEnc's process is faster than DupHunter's recompression.
- Throughput: While SimEnc's semantic-aware MLE can be a bottleneck, it still achieves a respectable average throughput of 135.2MB/s when partially decoding a layer.
- Clustering Effectiveness: SimEnc's novel similarity-preserving clustering algorithm using DBSCAN outperforms K-Means for semantic hash clustering, leading to better deduplication ratios.
Limitations and Future Work
- Semantic-aware MLE Throughput: The semantic-aware MLE step can be a performance bottleneck. The authors suggest that utilizing multiple GPUs for parallel inference could improve throughput.
- Security vs. Deduplication Trade-off: While SimEnc offers a benefit score to balance security and storage savings, there's an inherent trade-off. For extremely high security requirements, fewer unique keys might be generated, potentially impacting deduplication. The authors suggest that the key generation can be modified to favor unique keys for each chunk if needed.
- Dataset Quality: The effectiveness of semantic hashing relies on the quality of the training dataset. The authors note that the effectiveness hinges on dataset quality.
- Long-term Tracking: The paper mentions the importance of monitoring system evolution and performing timely rewarming or updates, suggesting this as an area for future work.
Summary updated