Dependency Analysis in Distributed Systems using Fault Injection: Application to Problem Determination in an e-commerce Environment

Saurabh Bagchi, Gautam Kar, Joseph L. HellersteinDSOM 2001 · OpenReview

Introduction

Modern distributed applications, particularly large-scale e-commerce platforms, are composed of a vast array of heterogeneous hardware and software components. When performance issues like high latency or service unavailability occur, identifying the root cause is a significant challenge. This paper presents a systematic approach to problem determination by automatically mapping dependencies through controlled fault injection.

The Challenge of Distributed Dependencies

Troubleshooting in modern environments faces two primary hurdles:

  1. Dynamic Complexity: Dependencies are not static; resources and service relationships shift based on system load and evolving service agreements.
  2. Limitations of Static Analysis: Configuration files are often outdated or incomplete, failing to capture the nuances of cross-domain interactions.

To address these, the authors propose moving from ad hoc troubleshooting to Active Dependency Discovery (ADD), a method that identifies which specific resource—be it a database table, a servlet, or a DNS server—is responsible for a Service Level Agreement (SLA) violation.

Active Dependency Discovery (ADD) Methodology

The ADD technique utilizes a fault-injection framework to map system relationships dynamically.

Dependency Modeling

The system is represented as a Directed Acyclic Graph (DAG). Nodes represent individual components, while edges signify dependencies. These dependencies are categorized by "strength": Strong (S), Medium (M), Weak (W), or Absent (A).

Dependency graph showing relationships between services with varying strengths (Strong, Medium, Weak).

Fault Dictionaries and Closeness Metrics

To bridge high-level symptoms with low-level triggers, the researchers use "Fault Dictionaries." These map observable failures (e.g., "Access Denied") to injectable faults (e.g., "Memory Corruption"). To optimize the search space, a "closeness" metric is used to identify a Suspect Transaction Set (STS)—transactions that share similar dependency profiles with the failing one.

Iterative Pruning

Once a Diagnosis Set (DS) of potential culprits is established, a fault injection campaign begins. By systematically perturbing suspect resources and observing if the resulting behavior mimics the original failure, the system "prunes" non-culprits from the list.

Experimental Validation

The methodology was validated in a prototype environment mimicking a standard three-tier e-commerce architecture.

Three-tier e-commerce architecture used for validating the ADD methodology.

The setup included:

  • Middleware: IBM WebSphere Application Server.
  • Workload: The TPC-W benchmark, simulating retail activities like browsing and purchasing.
  • Database: A backend containing tables such as Item, Author, and Order.

Key Insights and Results

The core innovation is the use of controlled "perturbation" to reveal causal relationships that simple monitoring might miss. The paper demonstrates this through a failure scenario where a "browse item" transaction fails.

  1. The initial suspect list is broad, including database tables, session pools, and DNS servers.
  2. By applying the closeness algorithm, the system compares the failing transaction to healthy ones, pruning resources they share.
  3. The suspect list is narrowed to just three components: the Item Table, a specific servlet, and the DNS server.
  4. Sequential fault injection (e.g., inducing lock contention) then isolates the unique root cause.

Limitations and Future Work

While effective, the method is invasive. Running diagnostic injections requires the system to be functional enough to support the routines. Furthermore, the reliance on Lightweight Fault Injectors (LWFIs) may require specific application instrumentation. Future work aims to address the scalability of concurrent injections in even larger, more complex systems.

Summary updated

BibTeX
@inproceedings{bagchi2001dependency,
  author = {Saurabh Bagchi and Gautam Kar and Joseph L. Hellerstein},
  title = {{Dependency Analysis in Distributed Systems using Fault Injection: Application to Problem Determination in an e-commerce Environment}},
  booktitle = {DSOM},
  year = {2001},
  url = {https://openreview.net/forum?id=gwyR4mv9KCp},
}
Figure