Feedback-based, Automated Failure Testing of Microservice-based Applications

Chengxu Cui, Guoquan Wu, Wei Chen, Jiaxing Zhu, Jun Wei

Abstract

Modern distributed applications are moving toward a microservice architecture, in which each service is developed and managed independently, and new features and updates are delivered continuously. A guiding principle of microservice architecture is that it must be built to anticipate and mitigate a variety of hardware and software failures. In order to test the fault handling capabilities of microservces, this paper presents IntelliFT, a feedback-based, automated failure testing technique for microservice based applications, which aims to expose the defects in the fault-handling logic quickly. The initial experimental result on a medium-size microservice benchmark system shows that the proposed approach is effective.

Introduction

In a microservice architecture, applications are composed of numerous independent services. While this modularity enhances scalability, it also introduces fragility; a single database timeout or network latency can trigger cascading failures across the platform. To address this, engineers employ Fault Injection Testing (FIT), or "Chaos Engineering," to proactively identify vulnerabilities. However, traditional methods like random injection (e.g., Chaos Monkey) are inefficient, and Lineage-Driven Fault Injection (LDFI) often treats requests in isolation, failing to leverage cross-request intelligence. This paper introduces IntelliFT, an automated testing framework that utilizes feedback from previous tests to optimize the discovery of microservice vulnerabilities.

Methodology

IntelliFT implements a multi-stage pipeline designed to streamline the fault injection process.

Service Call Graph Mapping

The process begins with pre-processing, where a proxy captures user requests and uses distributed tracing (Zipkin) to construct service call graphs. These graphs represent the dependencies between services for specific requests.

Figure 1: Service call graphs representing dependencies between microservices.

SAT-Based Modeling and Feedback Exploration

The call graphs are converted into logical formulas (CNF), and a SAT solver (Z3) identifies the minimal sets of service failures that could disrupt a request. The core innovation of IntelliFT is its feedback-based exploration. It assigns priority values to request classes, fault types (e.g., abort, delay), and specific services. As testing progresses, services or faults that reveal unique failures are prioritized for further exploration.

Figure 2: The IntelliFT algorithm for feedback-based failure testing.

Reduction Rules

To further optimize testing, IntelliFT employs two heuristic reduction rules:

  1. Propagation Rule: If a downstream fault is unhandled and propagates to the user, testing the same fault in upstream services is skipped, as they are already proven unable to handle the failure.
  2. Complexity Rule: If a single-point failure breaks the system, complex combinations involving that point are deferred until the primary issue is resolved.

Experimental Evaluation

The framework was evaluated using Trainticket, a medium-sized microservice benchmark simulating a railway ticketing system. The experimental environment utilized Istio for fault injection, Locust for traffic simulation, and Z3 as the SAT solver.

Key Results

The results demonstrate significant efficiency gains:

  • Search Space Reduction: While a native LDFI approach required 321 fault injections, IntelliFT reached the same conclusions with only 39 injections—a nearly 90% reduction.
  • Rule Effectiveness: The Propagation Rule (Rule 1) was particularly effective in pruning the search space for the benchmark application.

Conclusion

IntelliFT demonstrates that cross-request intelligence and heuristic reduction rules can significantly improve the efficiency of automated failure testing in microservice environments. Future work will focus on scaling the framework for industrial-grade deployments, expanding the variety of injected faults (e.g., data corruption), and validating the approach in live production environments.

Summary updated

Abstract

Modern distributed applications are moving toward a microservice architecture, in which each service is developed and managed independently, and new features and updates are delivered continuously. A guiding principle of microservice architecture is that it must be built to anticipate and mitigate a variety of hardware and software failures. In order to test the fault handling capabilities of microservces, this paper presents IntelliFT, a feedback-based, automated failure testing technique for microservice based applications, which aims to expose the defects in the fault-handling logic quickly. The initial experimental result on a medium-size microservice benchmark system shows that the proposed approach is effective.
BibTeX
@misc{cui2019feedback,
  author = {Chengxu Cui and Guoquan Wu and Wei Chen and Jiaxing Zhu and Jun Wei},
  title = {{Feedback-based, Automated Failure Testing of Microservice-based Applications}},
  year = {2019},
  eprint = {1908.06466},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.06466},
}
Figure