EdgePier: P2P-based Container Image Distribution in Edge Computing Environments

Soeren Becker, Florian Schmidt, Odej Kao

TL;DR

This paper introduces EdgePier, a decentralized peer-to-peer container registry designed for edge computing environments. EdgePier significantly speeds up container deployments by enabling nodes within an edge site to share image layers directly, bypassing slow cloud uplinks and reducing overall provisioning times by up to 65%.

What Problem Does This Paper Address?

Edge and fog computing environments are increasingly relying on containerization for lightweight and flexible application deployment. However, a critical bottleneck emerges during large-scale rollouts or updates: the centralized container registry. In edge scenarios, where devices are often geographically dispersed and connected via high-latency, low-bandwidth links to the cloud, this centralized registry becomes a performance bottleneck. This leads to longer deployment times, increased latency for applications like Function-as-a-Service (FaaS), and slower model updates for AI workloads. Existing solutions often involve complex, dedicated infrastructure that is not well-suited for the dynamic and resource-constrained nature of edge environments.

Methods

EdgePier tackles this challenge by implementing a fully decentralized, peer-to-peer (P2P) container registry. The core idea is to leverage the InterPlanetary File System (IPFS) as the underlying storage and distribution mechanism.

Here's how it works:

  • EdgePier Registry Service: This service runs on each node within an edge site. It exposes an HTTP API compatible with the official Docker registry API. This ensures non-intrusive integration, meaning existing container runtimes (like Docker or Kubernetes' kubelet) don't need modification.
  • IPFS Integration: When a Docker daemon requests an image, the EdgePier service first checks its local IPFS storage. If the image manifest and its constituent layer blobs are available, they are served directly.
  • P2P Distribution: If an image or its layers are not locally cached (a "cold start"), EdgePier utilizes IPFS to locate and download the necessary data from other peers within the edge site. IPFS, built on protocols like libp2p and Kademlia DHT, handles the discovery and efficient transfer of data blocks.
  • Decentralized Storage and Replication: Images are stored in IPFS, which uses content-addressable identifiers (CIDs). This ensures data integrity and deduplication. The system also incorporates a replication strategy using ipfs-cluster to maintain high availability of images across nodes in the edge site.

Datasets

Not discussed in the paper. The evaluation uses custom container images of varying sizes (100MB to 1GB) created for the experiments.

Key Insights and Ideas

  • Edge-Specific Bottleneck: The paper clearly articulates how the traditional centralized container registry model breaks down in the context of edge computing due to network limitations.
  • Leveraging IPFS for Decentralization: The ingenious use of IPFS as the backend for a container registry is a key innovation. IPFS's inherent P2P nature, content addressing, and block distribution mechanism are perfectly suited for this problem.
  • Non-Intrusive Deployment: The compatibility with the Docker registry API is crucial. It allows EdgePier to be adopted without modifying existing orchestration tools or container runtimes, significantly lowering the barrier to entry.
  • Bandwidth as a Key Differentiator: The evaluation highlights how EdgePier's performance advantage grows dramatically as the bandwidth to the cloud decreases, showcasing its effectiveness in real-world edge scenarios.
  • Network Load Reduction: Beyond just speed, EdgePier significantly reduces the load on the uplink connection to the cloud, freeing up bandwidth for other critical communications.

Key Results

The evaluation demonstrates significant improvements in container image provisioning times:

  • Up to 65% faster distribution: In scenarios with limited bandwidth (e.g., 20Mbps), EdgePier outperformed a baseline centralized registry by up to 65% in image distribution time.
  • Improved sequential and concurrent deployments: Both sequential (e.g., FaaS scaling) and concurrent (e.g., ML model updates) container startup scenarios showed benefits.
  • Reduced network utilization: EdgePier significantly offloads traffic from the cloud uplink, utilizing it only for the initial download of an image into the site, with subsequent distribution handled P2P.
  • Scalability with image size: While the baseline registry's pull time increased linearly with image size, EdgePier's increase was much slower due to layer sharing, especially for larger images.

Limitations and Future Work

  • Cold Start Overhead: In a "cold start" scenario where no image layers are cached on any node, the initial pull on EdgePier can be slightly slower than a baseline registry due to IPFS lookup overhead. However, this is quickly compensated by P2P sharing.
  • Cross-Site Exchange: The authors plan to investigate cross-site image exchange to enable model exchange between different edge sites in future work.
  • Further Optimization: While EdgePier shows strong results, there's always room for further optimization of IPFS integration and replication strategies.

Summary updated

Abstract

Edge and fog computing architectures utilize container technologies in order to offer a lightweight application deployment. Container images are stored in registry services and operated by orchestration platforms to download and start the respective applications on nodes of the infrastructure. During large application rollouts, the connection to the registry is prone to become a bottleneck, which results in longer provisioning times and deployment latencies. Previous work has mainly addressed this problem by proposing scalable registries, leveraging the BitTorrent protocol or distributed storage to host container images. However, for lightweight and dynamic edge environments the overhead of several dedicated components is not feasible in regard to its interference of the actual workload and is subject to failures due to the introduced complexity.In this paper we introduce a fully decentralized container registry called EdgePier, that can be deployed across edge sites and is able to decrease container deployment times by utilizing peer-to-peer connections between participating nodes. Image layers are shared without the need for further centralized orchestration entities. The conducted evaluation shows that the provisioning times are improved by up to 65% in comparison to a baseline registry, even with limited bandwidth to the cloud.
BibTeX
@inproceedings{becker2021edgepier,
  author = {Soeren Becker and Florian Schmidt and Odej Kao},
  title = {{EdgePier: P2P-based Container Image Distribution in Edge Computing Environments}},
  booktitle = {IPCCC},
  year = {2021},
  url = {https://openreview.net/forum?id=0aL8cdrLt5},
}
Figure