A Case Study on Runtime Verification of a Continuous Deployment Process

Shoma Ansai, Masaki Waga

TL;DR

This paper explores applying runtime verification to Continuous Deployment (CD) pipelines to ensure they meet critical timing constraints. By using a symbolic monitoring tool called SyMon, the researchers discovered that while FluxCD consistently detects new container images within ten minutes, it frequently misses a five-minute window—a vital insight for teams deploying time-sensitive security patches.

What Problem Does This Paper Address?

In modern DevOps, Continuous Deployment (CD) is the engine that keeps software fresh. We push code, a container image is built, and a controller like FluxCD or ArgoCD eventually pulls that image into production. However, this process is often asynchronous. Most CD controllers operate by periodically "polling" an image registry to see if anything is new.

This creates a "blind spot." If you push a critical security patch, how long does it actually take for your CD system to notice? If an error occurs in the polling logic, a vulnerability might sit in production far longer than intended. While we have plenty of tools to monitor if an application is running, we have surprisingly few tools that monitor whether the CD process itself is meeting its timing requirements. The authors aim to fill this gap by applying formal runtime verification to the deployment pipeline.

Methods

The researchers focused on a standard stack: GitHub Actions for builds, GitHub Container Registry (GHCR) for storage, and FluxCD for deployment onto Kubernetes. To monitor this, they employed Runtime Verification (RV)—a technique where a system's execution logs are checked against a formal specification in real-time.

The key components of their method included:

  • SyMon: A specialized tool for "symbolic monitoring." It handles "timed data words," which are essentially log entries that include both a timestamp and specific data (like a package name or version tag).
  • Event Mapping: they defined two primary events:
  • create: When an image is successfully pushed to GHCR (captured via GitHub webhooks).
  • fetch: When FluxCD’s logs show it has resolved the latest image tag from the registry.
  • Formal Specification: They wrote a high-level specification for SyMon that defines a "failure" as any create event that isn't followed by a matching fetch event within a specific timeout (e.g., 300 or 600 seconds).
  • Log Preprocessing: Since SyMon doesn't yet speak "raw JSON," the authors built a preprocessor to convert standard cloud logs into a simplified format compatible with symbolic automata.

Datasets

The study didn't use a public benchmark but rather "lived" data from a real CD environment.

  • Collection Period: Logs were collected over 15 days.
  • Volume: The dataset scaled from 12,758 entries (5 days) to 41,151 entries (15 days).
  • Events of Interest: Within the 5-day sample, there were 12 distinct create events (image pushes) used to test the timing constraints.

Key Insights and Ideas

  • The Polling Latency Gap: The most practical takeaway is that "eventual consistency" in CD has a measurable clock. In their specific configuration, FluxCD was not a "near-instant" responder.
  • Symbolic vs. Simple Monitoring: Unlike standard dashboard alerts that might just tell you "the service is up," symbolic monitoring allows you to track the relationship between events across different systems (Registry vs. CD Controller) while maintaining the context of specific data (ensuring the fetch matches the exact tag of the create).
  • Low Overhead for High Assurance: A common fear with formal verification is that it's too slow for production. This paper demonstrates that checking tens of thousands of logs takes less than half a second, making it perfectly viable for near-real-time monitoring.

Key Results

The researchers tested two different time thresholds for FluxCD's detection:

  1. The 5-Minute Window: FluxCD failed this threshold 5 times during the test period. It simply didn't poll or resolve the new image fast enough to meet a strict 300-second limit.
  2. The 10-Minute Window: FluxCD passed this threshold 100% of the time. In every instance, the new image was detected within 600 seconds.
  3. Performance: SyMon proved highly efficient. It processed 12,758 log entries in 360ms and 41,151 entries in 398ms. This sub-second execution time suggests that the monitoring adds negligible overhead to the DevOps pipeline.

Limitations and Future Work

The primary hurdle identified is log compatibility. Currently, SyMon cannot ingest JSON logs directly. The authors had to manually create a preprocessing step to extract package names, tags, and Unix timestamps. They acknowledge that this "implementation burden" is a barrier to wider adoption.

Future Work will focus on:

  • Updating SyMon to support JSON-formatted logs natively.
  • Reducing the manual effort required to set up the monitoring environment.
  • Developing a more "plug-and-play" practical monitoring system for engineers who aren't experts in formal methods.

Summary updated

Abstract

We report our experience in applying runtime monitoring to a FluxCD-based continuous deployment (CD) process. Our target system consists of GitHub Actions, GitHub Container Registry (GHCR), FluxCD, and an application running on Kubernetes. We monitored its logs using SyMon. In our setting, we regard a deployment update as detected when FluxCD's polling log resolves the latest image tag. Through the case study, we found that FluxCD did not always detect a new image within five minutes after it was pushed to GHCR, whereas it always did so within ten minutes in the collected logs. Moreover, our results show that SyMon is fast enough for near-real-time monitoring in our setting.
BibTeX
@misc{ansai2026case,
  author = {Shoma Ansai and Masaki Waga},
  title = {{A Case Study on Runtime Verification of a Continuous Deployment Process}},
  year = {2026},
  eprint = {2602.20598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20598},
}
Figure