FogDocker: Start Container Now, Fetch Image Later

Lorenzo Civolani, Guillaume Pierre, Paolo Bellavista

TL;DR

This paper introduces FogDocker, a system that significantly speeds up container deployment in resource-constrained environments like fog computing. By intelligently downloading only the essential parts of a Docker image first, FogDocker allows containers to start much faster, with the rest of the image fetched asynchronously in the background.

What Problem Does This Paper Address?

Deploying applications using containers, especially in fog computing environments, can be slow. Fog computing often relies on modest hardware like single-board computers, where network bandwidth can also be limited. The traditional Docker deployment process requires downloading the entire container image before the container can start, which can take minutes. This delay directly impacts the Quality of Experience for end-users who expect immediate service. The problem is exacerbated by the fact that many Docker images are large, but only a small fraction of their content is actually needed for a container to begin its useful work.

Methods

FogDocker tackles this problem by optimizing the container image deployment process. The core idea is to split the container image into two parts:

  • Essential Files (Base Layer): A minimal set of files required for the container to start and perform its initial operations.
  • Remaining Files: The rest of the image content, which can be downloaded later.

The process involves:

  1. Profiling: Using the fanotify API and the fatrace tool, FogDocker identifies the essential files accessed by a container during its startup phase. This is done by running the container with a developer-provided script that mimics typical usage.
  2. Base Layer Creation: A new "base layer" is created containing only these essential files, along with the necessary directory structure. This is achieved by exporting the original image, copying the identified essential files and their links, and then creating a new Dockerfile that uses this base layer.
  3. Modified Pull Operation: The Docker daemon is modified to prioritize downloading and installing this new base layer first. The other image layers are initially created but left empty.
  4. Asynchronous Download: Once the container starts using the base layer, the remaining image layers are downloaded and installed asynchronously in the background.
  5. Wrapper Library: To ensure correctness, a small wrapper library intercepts system calls like open() and fopen(). If a container tries to access a file that hasn't been downloaded yet but is part of the full image, the call is blocked until the file becomes available. This wrapper is transparent to the application.

Datasets

The paper evaluates FogDocker using four popular Docker images: httpd (Apache web server), nginx (web server), redis (key-value store), and a custom application called glife written in Go. These images were used to represent different types of applications and workloads. The evaluation was conducted on a Raspberry Pi 3 Model B+ (simulating a fog environment) and a powerful cloud server.

Key Insights and Ideas

  • The 6% Rule: The observation that containers typically access only a small fraction (around 6%) of their image content before starting useful work is the fundamental insight driving FogDocker. This suggests significant room for optimization.
  • Decoupling Startup from Full Download: The core innovation is decoupling the container's ability to start from the complete download of its image. This is a paradigm shift from the traditional Docker model.
  • OverlayFS Flexibility: FogDocker leverages the capabilities of OverlayFS, specifically its ability to handle runtime modifications to lower directories, even though this is an undocumented feature. This allows for the dynamic addition of content to image layers after a container has started.
  • Transparency is Key: The use of a wrapper library for system calls ensures that the optimization is transparent to the application running inside the container, requiring no modifications to the application code itself.
  • Performance Gains in Constrained Environments: The benefits of FogDocker are most pronounced in resource-constrained environments, particularly with limited network bandwidth, where reducing the initial download size has a dramatic impact.

Key Results

  • Reduced Deployment Time: FogDocker significantly reduces container deployment times. On single-board computers (fog scenario), speedups of 3x to 5x were observed for the time until the container could start. On powerful cloud servers, speedups of 1.8x to 3x were achieved.
  • Dramatic Speedup with Bandwidth Caps: In low-bandwidth conditions (e.g., 1 Mbps), the speedup increased dramatically, reaching up to 18x for the glife application.
  • Minimal Base Layer Size: The base layers created by FogDocker were significantly smaller than the original images, representing only 3% to 13% of the original size.
  • Modest Overhead: The wrapper library for system calls introduced a negligible overhead when the requested file already exists. When a file needs to be delayed, the wrapper adds a tolerable overhead of 7-8 ms for non-existent files that are not part of the image.
  • Slight Increase in Total Image Size: The total size of the FogDocker image on disk was slightly larger than the original due to the redundancy introduced by the base layer, but this overhead was considered modest.

Limitations and Future Work

  • OverlayFS Undocumented Behavior: FogDocker relies on an undocumented feature of OverlayFS. While discussions with developers suggest future versions will remain consistent, this is a potential point of concern.
  • Statically-Linked Executables: The current wrapper library assumes dynamically linked executables. For statically-linked executables, they would either need to be replaced with dynamically-linked alternatives or recompiled with the wrapper.
  • Re-introducing Integrity Checks: The current implementation disables layer integrity checks to allow for empty layers. The authors plan to reintroduce these checks at the end of the asynchronous download phase to ensure layer integrity.
  • More Sophisticated Profiling: For complex applications, more advanced profiling techniques or integration into DevOps workflows are suggested for future work.
  • Error Handling for Corrupt Layers: While the integrity check is disabled, the authors acknowledge that corrupt layers cannot be reliably detected. Future work could address this.

Summary updated

Abstract

Slow software deployment is an important issue in environments such as fog computing where this operation lies in the critical path of providing online services to the end users. The problem is even worse when the virtualized resources are made of modest machines such as single-board computers. This paper leverages the observation that, although Docker images are often very large, only a small fraction of their content is actually accessed by the containers during startup. We therefore propose to reorganize container images and download only the strictly necessary files before starting a container. The remaining image contents can then be downloaded asynchronously while the container is already running. Our performance evaluations show that FogDocker reduces container deployment times in the order of 3-5x on single-board computers and 2-3x on powerful servers, while incurring low runtime overhead and maintaining correctness even in the case a container accesses a file which was not downloaded yet.
BibTeX
@inproceedings{civolani2019fogdocker,
  author = {Lorenzo Civolani and Guillaume Pierre and Paolo Bellavista},
  title = {{FogDocker: Start Container Now, Fetch Image Later}},
  booktitle = {UCC},
  year = {2019},
  doi = {10.1145/3344341.3368811},
}

Loading related content...

Figure