Introduction
Modern distributed applications, particularly large-scale e-commerce platforms, are composed of a vast array of heterogeneous hardware and software components. When performance issues like high latency or service unavailability occur, identifying the root cause is a significant challenge. This paper presents a systematic approach to problem determination by automatically mapping dependencies through controlled fault injection.
The Challenge of Distributed Dependencies
Troubleshooting in modern environments faces two primary hurdles:
- Dynamic Complexity: Dependencies are not static; resources and service relationships shift based on system load and evolving service agreements.
- Limitations of Static Analysis: Configuration files are often outdated or incomplete, failing to capture the nuances of cross-domain interactions.
To address these, the authors propose moving from ad hoc troubleshooting to Active Dependency Discovery (ADD), a method that identifies which specific resource—be it a database table, a servlet, or a DNS server—is responsible for a Service Level Agreement (SLA) violation.
Active Dependency Discovery (ADD) Methodology
The ADD technique utilizes a fault-injection framework to map system relationships dynamically.
Dependency Modeling
The system is represented as a Directed Acyclic Graph (DAG). Nodes represent individual components, while edges signify dependencies. These dependencies are categorized by "strength": Strong (S), Medium (M), Weak (W), or Absent (A).
Fault Dictionaries and Closeness Metrics
To bridge high-level symptoms with low-level triggers, the researchers use "Fault Dictionaries." These map observable failures (e.g., "Access Denied") to injectable faults (e.g., "Memory Corruption"). To optimize the search space, a "closeness" metric is used to identify a Suspect Transaction Set (STS)—transactions that share similar dependency profiles with the failing one.
Iterative Pruning
Once a Diagnosis Set (DS) of potential culprits is established, a fault injection campaign begins. By systematically perturbing suspect resources and observing if the resulting behavior mimics the original failure, the system "prunes" non-culprits from the list.
Experimental Validation
The methodology was validated in a prototype environment mimicking a standard three-tier e-commerce architecture.
The setup included:
- Middleware: IBM WebSphere Application Server.
- Workload: The TPC-W benchmark, simulating retail activities like browsing and purchasing.
- Database: A backend containing tables such as
Item,Author, andOrder.
Key Insights and Results
The core innovation is the use of controlled "perturbation" to reveal causal relationships that simple monitoring might miss. The paper demonstrates this through a failure scenario where a "browse item" transaction fails.
- The initial suspect list is broad, including database tables, session pools, and DNS servers.
- By applying the closeness algorithm, the system compares the failing transaction to healthy ones, pruning resources they share.
- The suspect list is narrowed to just three components: the Item Table, a specific servlet, and the DNS server.
- Sequential fault injection (e.g., inducing lock contention) then isolates the unique root cause.
Limitations and Future Work
While effective, the method is invasive. Running diagnostic injections requires the system to be functional enough to support the routines. Furthermore, the reliance on Lightweight Fault Injectors (LWFIs) may require specific application instrumentation. Future work aims to address the scalability of concurrent injections in even larger, more complex systems.
Summary updated