Automated repair of cloud software depends on a diagnosis step: when something breaks, a root cause analysis method (an algorithm that points to the service responsible for a failure) decides where the system should intervene. Almost every public test of these methods uses synthetic failures, meaning faults injected by script into demo applications. Those scripts may leave fingerprints, such as a CPU spike that starts on an exact second, a fixed fault size, or a quiet background. A diagnosis method can score well by detecting the fingerprint rather than the failure. We propose to find such fingerprints automatically. We learn compact embeddings of the benchmark's telemetry, split them into a small set of readable "concepts" with a sparse autoencoder (a network that rewrites each embedding as a few active features out of many), and flag concepts that track the injection script more closely than they track the fault itself. We then measure which diagnosis methods succeed mainly on cases where those fingerprint concepts fire.
The claim can be tested. If the audit is right, the methods it flags as fingerprint-reliant should lose the most accuracy when the same faults are injected in a more realistic way. The audit should also recover fingerprints that we plant in the benchmark on purpose and never tell it about. If both hold, benchmark builders gain a cheap check that runs before release and names the specific script artifact to fix. If the audit only rediscovers artifacts that engineers already know to perturb, the honest conclusion is that a harder benchmark is enough, and the proposal says so.
Why this matters now
The standard open benchmark is RCAEval. It contains 735 injected failure cases across Online Boutique, Sock Shop and Train Ticket, the last with 64 services. The cases are split into a metric-focused set (RE1, 375 cases of CPU, memory and disk faults), a multi-source set (RE2, 270 cases with metrics, logs and traces), and a code-level set (RE3, 90 cases). The authors themselves leave subtle "gray failures" (intermittent, partial degradations) out of scope and never measure how well their synthetic cases stand in for production incidents.
The reported scores show a pattern consistent with sensitivity to how faults are generated. BARO reaches perfect accuracy (1.0) on disk faults but only 0.63 Avg@5 on network delay. Avg@5 is the average, over k from 1 to 5, of how often the true root cause appears among a method's top k guesses.
A sharper warning comes from multi-agent diagnosis. AgentTrace builds a causal graph over agent execution logs and reaches 94.9% Hit@1 on 550 synthetic failure scenarios. Positional features alone reach 87.3%, which raises the question of whether the injected bugs simply sit in predictable places.
In contrast, TraceDiag is trained and evaluated on 345 days of Microsoft Exchange production telemetry. It reports 82.1% precision in live use. That data is private, so the community cannot check synthetic results against it. Together these papers motivate the problem: current benchmark numbers may partly measure how well methods fit the injection scripts.
Core Claim
A sparse concept breakdown of learned telemetry embeddings can separate two kinds of feature. Some concepts describe the fault, for example memory pressure building on one service. Others describe the injection script, for example an onset that is locked to the script's start time with near-zero spread across cases.
For each diagnosis method, we define an artifact reliance score from how that method's success varies with the strength of these script-tracking concepts. The central claim is that this score, computed only on the original benchmark, predicts how much each method degrades under a more realistic generator. It should do so better than simpler audits.
Research Program
The method borrows from two interpretability papers.
- Concept extraction. SpLiCE rewrites CLIP embeddings as non-negative sparse combinations of a concept dictionary, with 10 to 30 active concepts out of 15,000. It uses those concepts to expose dataset biases while keeping zero-shot accuracy; on CIFAR100 it reports 0.768 against CLIP's 0.750.
- Concept labelling and shifts. The ESM-2 sparse autoencoder work trains sparse autoencoders on protein-model embeddings and aligns the features with curated annotations. It then reads mutation effects as drops in concept activation. Its sparse features match the raw backbone on ProteinGym (Spearman 0.719 against 0.715).
We transfer the recipe but change its target. The concepts are used to audit the data, not to diagnose faults directly.
Encoding the telemetry. We encode per-service metric windows with the contrastive encoder of RUN. RUN trains on positive pairs taken from the same timestamp in different context windows. This avoids the "periodicity trap," in which ordinary contrastive learning treats similar daily cycles on different days as dissimilar.
Learning the concepts. We train a sparse autoencoder on pooled embeddings from normal and faulty windows across all three RCAEval systems. Each learned concept is labelled in the ESM-2 style, by correlating it with two things: interpretable window statistics (level, slope, onset sharpness, variance, cross-service correlation) and the injection metadata (fault type, target service, injection timestamp, configured magnitude).
Flagging script-tracking concepts. A concept is flagged when injection metadata explains its activation beyond what the fault semantics do. Examples are activation timed to the script's start with sub-second consistency, or magnitude that is nearly constant across cases of one fault type.
Scoring each method's reliance. Many diagnosis methods, such as BARO and CIRCA, are not neural networks, so we measure reliance from outcomes instead of gradients. For method $m$ and flagged concept $c$, let $R_{m}$ be the drop in method success between cases where $c$ fires strongly (top third of activation) and cases where it fires weakly (bottom third). We take the maximum over flagged concepts, with bootstrap confidence intervals.
Building the reference generator. We build a perturbed generator on the same Kubernetes deployments. It uses the same fault families but adds gradual ramps instead of steps, intermittent faults, randomized magnitudes, background load changes that overlap the fault, and occasional pairs of simultaneous faults. This is exactly the gray-failure regime RCAEval left open.
Where a production partner can supply even a handful of verified incidents, we will add them as a third reference. We do not assume such access.
Baselines
Naive reference. The do-nothing option assumes the benchmark has no artifacts and predicts that methods keep their original ranking and scores under the perturbed generator. It is scored by rank correlation between original and perturbed performance. The strongest reported diagnosis figure it anchors on is TraceRCA's overall Avg@5 of 0.77 on RCAEval (RCAEval). If rankings barely move, there is nothing to audit and the project stops early.
Discriminator baseline. This is the strongest existing competitor. It is a classifier two-sample test: a gradient-boosted classifier trained to tell original windows from perturbed ones, with feature attributions on the raw window statistics used to name the differences. It is strong because it directly measures distribution gap and gives feature-level explanations. It gets the same windows, the same statistics, and the same tuning budget as our pipeline, with the same number of hyperparameter trials. No reported result was found on Lacuna for a discriminator audit of RCA benchmarks. Our method beats it if its reliance scores predict per-method degradation with higher rank correlation. A second route to beating it is to match that correlation while using only the original benchmark, since the discriminator needs the perturbed data to exist first.
Checklist baseline. An expert computes a fixed list of suspected artifacts (onset sharpness, magnitude spread, timestamp alignment) and scores reliance with the same $R_m$ procedure. This is the fair version of "just check the obvious things." We beat it only if the audit finds predictive artifacts that are not on the list.
Ablations. Two ablations isolate the key ingredient. The first replaces the sparse concepts with the top principal components of the same dense embeddings. The second trains the sparse autoencoder on the raw statistics without a learned encoder. If the principal-component version predicts degradation just as well, sparsity adds nothing beyond a readable label, and we will report that.
Why not just…?
Why not just build the harder benchmark and rerun every method? This is the real threat, because direct stress testing is the ground truth our audit is scored against. The audit is only worth having if it finds artifacts a benchmark builder would not have thought to perturb. We test this in two ways. First, we plant artifacts that are absent from both the perturbation list and the checklist, such as a faint fixed-period marker in one metric during injections, or fault magnitudes correlated with target service. Second, we hold out whole perturbation families when designing the audit. If recovery of planted artifacts is poor, or the audit adds nothing on held-out families, the right contribution is the gray-failure benchmark itself, and we will release it as such.
Why not just train a discriminator between synthetic and more realistic data? A discriminator needs a second dataset, and for production incidents that dataset is usually private, as with TraceDiag's Exchange logs. It also finds every difference between the datasets, not the ones diagnosis methods depend on. The discriminator baseline above tests whether this matters in practice.
Why not just ask an LLM to read traces or injection scripts? On diagnosis itself, AgentTrace reports GPT-4 at 68.5% Hit@1 against its own 94.9%, often picking symptom nodes rather than origins. Reading the scripts can list what was injected, but not which regularities a given method actually uses. We would include an LLM-generated artifact list as an extra checklist variant.
Why not just use the dense contrastive embeddings? RUN's embeddings are already useful for detecting faults, but dense directions do not map to nameable script behaviours. The principal-component ablation measures whether naming them changes predictive power.
Proposed Experiments
Experiment 1: predicting degradation. Using RE1 and RE2 across all three systems, we compute $R_m$ for BARO, CIRCA, CausalRCA, TraceRCA, MicroRank, PDiagnose and RUN, for each fault type. This gives roughly 50 method-by-fault-type cells. We then run all methods on the perturbed generator and measure the drop in Avg@5. The decisive check is the Spearman rank correlation between $R_m$ and that drop across cells, compared against every baseline. A clear, bootstrap-significant advantage over the discriminator and the checklist supports the claim. A tie with the checklist means the artifacts are the obvious ones.
Experiment 2: planted artifacts. We measure precision and recall of the flagging rule on the planted artifacts, across several planting strengths. This shows whether the audit finds fingerprints nobody listed in advance.
Experiment 3: repair loop. For the top flagged artifact, we remove it from the original generator and check two things. The flagged concept's activation should collapse, and the originally over-scored methods should lose accuracy while the others hold steady. This closes the loop from diagnosis of the benchmark to a concrete fix for its generator.
Transfer check. As a secondary check, we apply the same audit to AgentTrace's 550 synthetic scenarios, using embeddings of trace steps. The question is whether the 87.3% positional result reflects bugs being injected at predictable positions in the trace.
References
- RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data
- AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
- TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice Systems
- Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
- Decoding Loss-of-Function Variants with Sparse Concept Features of ESM-2
- Root Cause Analysis in Microservice Using Neural Granger Causal Discovery