Spending the Telemetry Budget on Deviations: Spiking Encoders for Gray Failure Detection

Gray failures are degradations that cloud health checks miss: a node still answers pings, but requests through it are slow, dropped, or corrupted. Catching them requires fine-grained telemetry from many vantage points, and that telemetry is expensive to ship and process. Production systems either sample it coarsely, or point the budget at busy components, which leaves quiet components unwatched. This proposal borrows one idea from neuromorphic computing. A spiking neuron accumulates its input over time with a slow leak, so small but persistent deviations add up, and it emits a discrete event only when the running total crosses a threshold. Placed at each telemetry source, such neurons report almost nothing while the system is healthy and report events when it drifts. A small spiking network downstream then flags the defining signature of a gray failure: the application layer is reporting trouble while the infrastructure layer is not.

The claim is narrow and testable. At a fixed telemetry budget, measured in events per node per second, learned leaky-integrating encoders should detect and localize subtle faults earlier and more reliably than periodic sampling, send-on-change compression, or per-metric statistical alarms feeding the same detector. The advantage should be largest for faults on low-traffic services. Energy enters as an accounting lens, not a headline. We count transmitted events and synaptic operations, because in monitoring the cost that matters is shipping and ingesting data, not the arithmetic of the detector.

Why this gap, and why the obvious version is the wrong one

The problem is well documented on the systems side. Gray Failure: The Achilles' Heel of Cloud-Scale Systems names the cause differential observability: different observers see the same component differently. It also warns that probing a degraded node can push it over the edge. This motivates our cross-plane mismatch detector.

The cost of watching is concrete. Mycroft gets collective-communication tracing under 1% overhead by switching from event logs to state logs and sampling one rank per group, yet it still ingests about 3 TB of logs a day. The profilers it replaced cut bandwidth by a third or more. NTP-INT halves network telemetry probes by spending them only on switches predicted to exceed 80% load. Its authors acknowledge the consequence: silent drops on lightly loaded switches go unwatched, the same failure class that 007 hunts for. Both papers motivate the proposal. Each controls cost by choosing where to look. We control it by choosing when to speak, so the budget follows deviation rather than load.

MicroRes shows that degradation spreads from internal metrics to user-facing ones. It measures this spread offline with dynamic time warping, at about 302 seconds per case. We want the same cross-layer signal online. Compound AI System Reliability supplies the stakes: silent degradations were 51% of 150 incidents and took 4.2 days on average to detect, versus 12 minutes for crashes.

The neuromorphic papers supply mechanisms but no systems validation. Encrypted internet traffic classification using a supervised spiking neural network shows that a 100-neuron spiking network classifies traffic from packet timing alone, with 95.9% accuracy on 14 classes, but only as offline classification of fixed windows. SH²RFSSM uses spiking oscillator neurons that remain stable over 50,000-step sequences. ChronoSpike runs spiking graph models at 83–88% activation sparsity on citation and transaction graphs. Neither has seen telemetry, call graphs, or a budget on communication. Work on LIF coincidence detection, such as Calcium current improves coincidence detection of the LIF model, supplies the primitive we reuse for cross-plane mismatch.

The obvious combination is "run a spiking anomaly detector on telemetry to save energy." It deserves rejection. A Z-score detector on a CPU already costs almost nothing, and the 69× energy figure for SH²RFSSM over LinOSS concerns model arithmetic, which is not where monitoring spends its resources. What we propose differs in where the spiking dynamics sit: at the source, as the compression and decision layer, trained end to end against a detection objective under an explicit event budget.

Core mechanism

Each scalar metric stream $x_t$ (a latency percentile, error rate, queue depth, or CPU counter) passes through a two-sided leaky integrator:

$$u_t = \lambda\, u_{t-1} + \frac{x_t - \mu_t}{\sigma_t}, \qquad s_t = \mathrm{sign}(u_t)\,\mathbb{1}[|u_t| > \theta], \quad u_t \leftarrow 0 \text{ if } s_t \neq 0.$$

Here $\mu_t$ and $\sigma_t$ are slowly updated running estimates of the metric's normal level and spread, and $\lambda \in (0,1)$ is the leak. $\theta$ is the firing threshold, and $s_t \in \{-1, 0, +1\}$ is the event actually transmitted. Only nonzero $s_t$ leave the node.

With $\lambda$ close to 1, this encoder is an exponentially forgetting CUSUM, the classic sequential test for detecting a sustained shift in a mean, with a reset after each report. We say this openly because it defines the strongest baseline. The spiking formulation adds one thing: $\lambda$ and $\theta$ become per-metric, per-service parameters. We learn them jointly with the downstream detector using surrogate gradients through the threshold, minimizing

$$\mathcal{L} = \mathcal{L}_{\text{detect}} + \beta\, \mathbb{E}[\,\|s\|_0\,] .$$

$\mathcal{L}_{\text{detect}}$ combines per-window fault detection with root-cause ranking. $\mathbb{E}[\|s\|_0]$ is the expected number of transmitted events per node per second. Sweeping $\beta$ traces a curve of detection quality against budget.

Downstream, events arrive at a two-stage spiking network. The first stage places, for each service, LIF coincidence units with short time constants. These units receive excitation from that service's application-plane events (caller-observed latency and errors from traces) and inhibition from its infrastructure-plane events (host and container counters). A unit fires when the application plane is deviating and the infrastructure plane is quiet, which is differential observability turned into a circuit. The second stage passes these spikes along the call graph extracted from traces, using a ChronoSpike-style spiking graph layer. It ranks candidate root causes, so that a fault propagating downstream is attributed to the furthest-upstream mismatching service.

Research Program

The work has three parts.

First, we build a graded-intensity gray-failure corpus. The open microservice systems used by RCAEval are Online Boutique, Sock Shop, and the 64-service Train Ticket. On these we inject RCAEval's fault classes at several intensities: packet loss from 0.5% to 5%, added delay from 5 to 50 ms, slow memory leaks, and socket errors. The low end is designed to leave infrastructure health checks green. Faults are placed on both high- and low-traffic services so the blind-spot question can be measured directly. We also keep RCAEval's original RE1 and RE2 cases unchanged, so results connect to published numbers.

Second, we implement the encoders and the spiking detector in software, with surrogate-gradient training. We report the full curve of detection quality against event budget, not a single operating point.

Third, we do the cost accounting. We count transmitted events and bytes, ingest volume, and synaptic accumulate operations. From the operation counts we estimate arithmetic energy using the 45 nm figures quoted in SpikingBERT: 0.9 pJ per accumulate versus 4.6 pJ per multiply-accumulate. We label these as estimates. Deployment on SmartNICs or neuromorphic chips is out of scope; the hardware results summarized in Energy Efficient Computing with Spiking Neural Networks motivate that later step but are not claimed here.

Baselines

The naive reference is the current production default. Prometheus-style periodic scraping at a fixed interval feeds a dense detector. We run it at full rate as an upper reference for accuracy and cost. We also subsample it to each matched event budget, which is the fair low-budget comparison. Beating the subsampled version at equal budget is the minimum bar. Matching the full-rate version at a tenth or less of its traffic is the target.

The strongest existing root-cause methods are BARO and TraceRCA, as evaluated in RCAEval. They receive full-rate metrics and traces. On Train Ticket RE2, BARO reports Avg@5 of 1.0 on disk faults but 0.63 on network delay faults, and TraceRCA reports an overall Avg@5 of 0.77. We rerun both on our graded corpus with their published configurations and the same tuning budget we give our method. On the original RE1 and RE2 cases, success means matching TraceRCA's Avg@5 within noise while transmitting a small fraction of its input. On the low-intensity cases, success means beating it outright on detection delay.

The strongest encoder competitors are three hand-designed source-side schemes: send-on-delta with a deadband, per-metric CUSUM with alarm forwarding, and Aegis-style Z-score alarms (Evolution of Aegis). Each feeds the identical downstream detector, and each has its thresholds tuned on the same validation split to hit the same event budgets.

Two ablations isolate the key ingredients. Replacing the spiking downstream network with a dense GRU plus graph network on the same events tests whether the spiking detector matters or only the spiking encoder. Removing the inhibitory infrastructure input from the coincidence units tests whether modeling differential observability explicitly matters.

Why not just…?

Why not just use per-metric CUSUM or send-on-delta at the source? This is the real threat, because our encoder contains CUSUM as a special case. The experiment settles it. If hand-tuned CUSUM at matched budget comes within noise of the learned encoders on detection delay and Avg@5, then end-to-end learning of leaks and thresholds adds nothing. In that case we will report CUSUM encoding as the recommended design, and the remaining contribution is the cross-plane coincidence layer. We expect learning to help mainly on low-traffic services and multi-metric faults, where one global threshold either floods the budget or misses the shift. That expectation is a hypothesis, not established evidence.

Why not just run Z-score outlier detection at each node? Aegis did exactly this in its first phase. It cut idle time by 71% but confused host-side faults with network faults and needed offline diagnosis. Only instrumenting the communication runtime brought the reduction to 91% (Evolution of Aegis). Isolated per-metric alarms lack the cross-layer, cross-service context that localization needs, and our Z-score baseline and coincidence ablation measure that gap directly.

Why not just keep full telemetry? Detector compute is cheap. We agree that detector compute is cheap, and we make no datacenter-energy claim for the detector. The expensive parts are shipping and ingesting telemetry, and probing nodes that are already fragile: Mycroft's 3 TB a day, profilers that cost a third of bandwidth (Mycroft), and NTP-INT's decision to leave low-load switches unwatched (NTP-INT). The full-rate baseline shows how much accuracy, if any, the budgeted encoders give up.

Why not just use a dense model on the same sparse events? Our dense-detector ablation is exactly that test. If the dense model matches, the honest conclusion is that the encoding carries the benefit and the spiking downstream layer is an implementation choice. That choice would then be justified only by event-driven deployment on a SmartNIC, which we flag as future work.

Proposed Experiments

The decisive check is the curve of detection quality against event budget on the low-intensity faults placed on low-traffic Train Ticket services. Quality is measured as recall at a fixed false-alarm rate, median detection delay, and Avg@5 localization. Learned spiking encoders must lie above subsampled scraping and above tuned CUSUM over at least a decade of budget. If they do not, the learning claim fails.

The second check asks whether the coincidence units add value specifically on cases where infrastructure metrics stay within normal bounds. We identify these cases afterwards by checking that no host counter exceeds its validation range. On exactly these cases, removing inhibition should collapse precision.

Third, on the unmodified RCAEval RE1 and RE2 cases, we report Avg@5 next to the published BARO and TraceRCA numbers, together with the fraction of their input volume our method transmitted.

As a stress test of the low-traffic claim, we inject silent drops on lightly loaded switches in the HPCC NS-3 setup used by NTP-INT. We compare drop detection under deviation-driven encoding with detection under load-driven probe allocation at equal probe volume.

No reported result was found on Lacuna for any spiking or event-driven encoder on RCAEval or on gray-failure injection, so every number in this proposal's favor must come from these runs.