Stress-Probe Surrogates: Predicting Weeks-Long Switch Health from Hours of Testing

Programmable networks now let a central controller push new packet-processing code into switches while they run. That code is tested for minutes and then left in production for months. When a new version slowly leaks memory, lets a shared buffer drift out of sync, or degrades under constant rule updates, nobody finds out until it fails. Clinical trials and economics face the same problem under the name "limited follow-up": the short trial looks good, and the long-term outcome is unknown. Those fields have a tool for it. A surrogate index is a model, fitted on past cases where both early measurements and long-term outcomes were recorded, that maps the early measurements to a predicted long-term outcome. We propose to bring that tool to network software, with one change that only a programmable network permits. Because the controller can reprogram the forwarding hardware, it does not have to wait for weak early signals to appear. During the short test it can apply designed stress probes that force hidden, slowly accumulating damage to show up.

The claim in two sentences: a surrogate index trained on past long soak runs of switch software can predict the two-week effect of a new controller-deployed configuration from six hours of telemetry. Its accuracy improves substantially when those six hours include controller-scheduled probes, such as bursts of rule churn, packet-size sweeps, and redeployment of the processing code, whose recovery curves measure hidden aging directly. The first half applies an existing method to a new domain. The second half is the new ingredient, and the experiments are built to isolate it.

Core Claim

Separating the control plane (the software that decides forwarding policy) from the data plane (the machinery that forwards packets) creates two properties that clinical surrogate work lacks. The first is that the "treatment" is fully known and repeatable, since it is a software configuration the controller installs. The second is that the controller can run interventions on the system during the observation window. The surrogate-index literature treats surrogates as passively observed. The Surrogate Index combines many short-term proxies into a predicted long-term outcome. On the GAIN job-training experiment it recovered a 9-year employment effect of 6.1% against a true 6.4% from 1.5 years of data, with 35% smaller standard errors than waiting for the noisy outcome. Long-Term Effect Estimation with Surrogate Representation relaxes the assumption that treatment acts only through the observed surrogates. It learns balanced recurrent representations of the short-term trajectory.

Both methods can only use whatever early signals the world happens to produce. Their main failure mode is the "leaky" surrogate, where treatment reaches the outcome by a path the early measurements never see. A slow memory leak in a packet-processing program is a leak of exactly this kind: throughput and latency counters look normal for hours. Our hypothesis is that short, controller-scheduled stress probes can turn such hidden paths into observed ones. A probe might be 10,000 rule insertions followed by a measured recovery, or a forced restart of the processing program with the ring-buffer state checked afterward. The analogy is a cardiac stress test, which exposes damage that a resting measurement misses. The clinical evidence that multi-dimensional mechanical markers beat single markers motivates the design. Sohal et al. found that a 16-segment volume-change dyssynchrony index predicted reverse remodeling where the single ECG marker failed. Yang et al. raised AF recurrence prediction from AUC 0.703 (BNP alone) to 0.814 by adding left-atrial function measures.

Formally, let $W$ be a configuration (treatment), $S$ the vector of passive short-term telemetry, $P$ the vector of probe-response features, and $Y$ the long-term outcome (for example, day-14 throughput retention). We fit, on a historical cohort of long soak runs,

$$\hat{h}(s, p) = \mathbb{E}[Y \mid S = s, P = p],$$

and estimate the long-term effect of a new configuration $w$ against an incumbent $w_0$ from short runs alone as

$$\hat{\tau} = \frac{1}{n_w}\sum_{i: W_i = w}\hat{h}(S_i, P_i) - \frac{1}{n_{0}}\sum_{i: W_i = w_0}\hat{h}(S_i, P_i),$$

where $n_w$ and $n_0$ are the numbers of short-run instances assigned to each configuration. The testable prediction is that including $P$ reduces the error of $\hat{\tau}$ most for configurations whose defects are hidden when the system is at rest.

Research Program

The platform is the Action Program architecture of AP-SDN. It extends Open vSwitch so that a Ryu controller can compile and push standalone packet-processing binaries. Packets are handed to these binaries through shared-memory ring buffers. We chose it because the AP-SDN authors themselves list the failure modes it leaves unexamined: long-running user-space processes in the forwarding path, synchronization of the shared-memory queues, and no lifecycle management for stale programs. Its evaluation consists entirely of ping and short iperf3 runs. It reports a 26.1% average throughput drop at small packet sizes, parity at 1024 bytes, and 1.2 ms of extra deployment delay. Nothing is known about day 14.

We will build a soak-test cohort. It will contain several hundred switch instances, run as software switches on commodity servers or VMs. Each instance runs one configuration for 14 days under replayed traffic with a controlled mix of packet sizes. A configuration combines an action-program implementation, a controller rule-churn rate, and a redeployment schedule. The implementations are the AP-SDN transparent-encryption program and variants of it. Some variants are ordinary independent reimplementations. Others contain seeded defects: slow leaks, lock contention on the ring buffer, and unbounded per-flow state. We will also write several realistic "unseeded" versions without planting bugs, so that the benchmark does not reward detecting only the defects we planted.

The long-term outcomes, measured at day 14, are throughput retention relative to hour 1, 99th-percentile latency, resident memory, and whether a crash or hang occurred. The passive surrogates are the first six hours of data-plane counters (throughput, drops, queue occupancy, ring fill levels) and control-plane counters (flow-modification rates, controller CPU, rule-install latency). The probe features are recovery times, and changes in memory and throughput, after a fixed battery of about 20 minutes of controller-scheduled probes embedded in the six hours. The probe battery is itself a design variable. We will compare a hand-designed battery against one chosen greedily for how much it adds to predictive value on the training cohort.

On this cohort we fit the passive surrogate index, the LTEE recurrent representation model, and both again with probe features added. Evaluation holds out entire implementation families. The model must predict for code it has never seen, which is the operational question when a new version ships. It also targets the time-drift problem that LTEE names as the main open issue. For each held-out new-versus-incumbent pair we randomize instances between the two versions. We then estimate $\hat{\tau}$ from six hours of data and compare it against the true day-14 difference.

Baselines

The naive reference point is to trust the short test. We use the six-hour mean of the outcome metric itself, for example hour-6 throughput relative to hour 1, as the prediction for day 14. This is what the AP-SDN evaluation and most programmable-network papers implicitly do, and it defines "no follow-up correction." On the same held-out pairs we also report "wait-and-see", the true 14-day measurement. It is the ground truth, and its run-to-run variance gives the noise floor that any estimator is compared against.

The second baseline is trend extrapolation. We fit a per-instance linear or log-linear model to memory and throughput over the six hours and project it to day 14. This is the standard software-aging practice and the most natural thing an operator would try. It uses the same six hours of data and no historical cohort. If this baseline matches the learned models, the surrogate machinery is unnecessary. No reported result was found on Lacuna for trend extrapolation on switch software.

The strongest existing methods are the passive Surrogate Index and LTEE, trained on the same cohort, feature budget, and hyperparameter search budget as the probe-augmented models. The best reported figure for the surrogate index is its GAIN result: 6.1% estimated against 6.4% true for 9-year employment, with a 35% reduction in standard errors. LTEE reports lower absolute ATE error than surrogate-index and counterfactual-outcome baselines on IHDP and News. No reported absolute ATE numbers for LTEE were found on Lacuna, and no reported result was found on Lacuna for either method on network data. As a sanity check we will first reproduce LTEE's IHDP ordering to confirm our implementation.

The key ablation compares probe-augmented and passive models while matching observation time. The passive model gets the same six hours, including the 20 minutes the probes would have used. It also gets the same number of features, with the probe features replaced by the same number of extra passive statistics. This isolates whether the gain comes from intervening or merely from having more features.

A second ablation removes all control-plane-side surrogates. This tests whether the separation matters for measurement too: we expect rule churn and redeployment behavior to carry direct paths that data-plane counters alone miss.

The proposal wins if probe-augmented estimates cut the absolute error of the day-14 effect by at least a third relative to the best passive model on held-out families, with calibrated 90% intervals, and if they rank new-versus-incumbent pairs correctly more often than trend extrapolation.

Why not just…?

Why not just run accelerated stress tests, at ten times the load for a shorter time? This is the most serious threat, because our probes are a form of stress. Accelerated testing assumes that damage scales with load. It fails for defects triggered by elapsed time, by rare events, or by timing interactions, such as leaks tied to flow-table expiry timers or ring desynchronization that shows up only during redeployment. We include a 6-hour accelerated-load run as a direct competitor and report per-defect-class results. If accelerated load matches probes across classes, we will report that the useful ingredient is stress alone and that the calibrated mapping to long-term outcomes is what the surrogate model adds.

Why not just extrapolate the memory trend? Extrapolation works for monotone linear leaks and should do well on that seeded class. It has nothing to say about hangs, about throughput cliffs that come after stable plateaus, or about effects that depend on the traffic mix. The multi-marker evidence from Sohal et al. and Yang et al. suggests that single-trajectory signals miss these cases. The per-class breakdown will show exactly where extrapolation holds up and where it fails.

Why not just use the passive surrogate index as-is? The leaky-surrogate problem is the known failure of that method. Its own authors and the LTEE authors identify effects that bypass the observed proxies. The matched-time ablation decides this question directly. If the passive models are already accurate, the contribution shrinks to a new benchmark plus a negative result about probes, and we would report it that way.

Why not just wait two weeks? For a single deployment, waiting is feasible. Controllers, however, push configuration changes continually. AP-SDN notes that compiling and distributing binaries is already a scaling bottleneck, and holding every change for 14 days stalls that pipeline. Waiting is also not free of noise: the GAIN result showed that surrogate estimates can be more precise than the long-horizon outcome itself. We will test whether that holds here by comparing the variance of $\hat{\tau}$ with the between-run variance of the true day-14 differences.

Proposed Experiments

The decisive experiment is the held-out-family effect estimate described above. It reports absolute error of $\hat{\tau}$, interval coverage, pairwise ranking accuracy, and AUC for crash-by-day-14. All results are broken down by defect class: linear leak, nonlinear or cliff, contention, unseeded. The central prediction is that probes help most on the cliff, contention, and unseeded classes and least on linear leaks. If the gains were uniform across classes, the likelier explanation would be extra features rather than revealed hidden state.

A second check measures how the error of $\hat{\tau}$ changes as the short window shrinks from six hours to one hour, with and without probes. It tells operators how much test time they can save.

A third check tests transfer under traffic drift. We train on one traffic replay and evaluate on another with a shifted packet-size mix, using the AP-SDN finding that small packets carry the largest penalty as the stress axis.

If the method holds up, the same design carries over naturally to two other systems. One is cell on/off policies of the kind in the aerial O-RAN deployment, whose 40% daily energy saving was measured over simulated 24-hour cycles, and whose platform degrades physically in ways those cycles never exercise. The other is in-network dropping as in Octopus, which was evaluated on traces seconds to minutes long, and whose authors warn that long-lived TCP cross-traffic may starve. We treat both as follow-ups, not as part of this proposal's claim.