Attacker-Side Adversarial Training for Intrusion Detectors at a Fixed False-Alarm Budget

Adversarial training teaches a classifier to resist small worst-case input changes. It does this by training on deliberately perturbed copies of the data. The usual price is lower accuracy on ordinary, unperturbed inputs. In network intrusion detection that price is unusually harsh. Benign traffic makes up 98% to more than 99.99% of flows, so a one-point rise in the false-alarm rate buries analysts in alerts. Meanwhile the rarest attacks, such as privilege escalation or Heartbleed, have only a handful of training examples.

Most of what gets ported from vision makes three choices that do not fit this setting. It perturbs every class equally. It defends the default 0.5 decision boundary. It reports overall accuracy. A real intruder does none of these things. Only malicious flows are disguised, only in fields the attacker can change without breaking the connection, and the disguise only has to beat the alarm threshold actually deployed. That threshold is usually set so that roughly 1 benign flow in 1,000 to 10,000 trips it.

We propose to train against that actual threat. Perturbations would be applied only to attack flows and confined to changes that keep the traffic functional. The worst case would be searched relative to a threshold that is recomputed during training from benign traffic so that it holds a target false-alarm rate. Success would be judged by robust attack recall at that false-alarm rate.

Core Claim

The claim is that much of the robustness–accuracy trade-off seen when adversarial training is applied to intrusion detectors comes from a mismatched threat model and the wrong operating point. It is not inherent.

Hu et al.'s analysis gives a reason to expect this. In Understanding the Impact of Adversarial Robustness on Accuracy Disparity they analyse linear binary classifiers and split the clean-accuracy damage into two parts. One is an "intrinsic" change in the boundary's direction. The other is an "imbalance" effect: a shrinkage of the classifier's norm that grows with the class ratio and widens the gap between majority and minority classes. In a linear model, a norm shrinkage at a fixed bias moves where the boundary sits.

Our design takes over that placement job. The threshold is pinned to a quantile of benign scores, the score below which a chosen fraction of benign flows fall. Pinning the threshold that way should absorb most of the imbalance effect and leave mainly the directional cost. That cost should itself be smaller, because benign points are never pushed toward the attack region.

This yields a testable prediction. Under symmetric adversarial training, the drop in clean recall at a fixed false-alarm rate should grow with the imbalance ratio. Under our attacker-only, threshold-aware training, it should stay roughly flat. Hu et al.'s proof covers only linear models with equal class covariances, which is exactly what network data violates. So this is a hypothesis to test, not a guarantee.

Concretely, for a detector score $f_\theta(x)$ (higher means more likely an attack) we train

$$\min_\theta \; \mathbb{E}_{x\sim\text{benign}}\big[\ell(f_\theta(x),0)\big] + \lambda\,\mathbb{E}_{x\sim\text{attack}}\Big[\max_{\delta\in\Delta(x)} \ell_{\tau}\big(f_\theta(x+\delta)\big)\Big].$$

The symbols are as follows:

  • $\ell$ is cross-entropy.
  • $\Delta(x)$ is the set of changes an attacker can make to flow $x$ without breaking it.
  • $\tau$ is the $(1-\alpha)$ quantile of benign scores at the current training step, where $\alpha$ is the target false-alarm rate. It is re-estimated each epoch on a held-out benign pool.
  • $\ell_\tau$ is a hinge loss that penalises an attack flow whose perturbed score falls below $\tau$ plus a margin.
  • $\lambda$ weights the attack term.

The inner search therefore attacks the deployed alarm line rather than the argmax boundary. At $\alpha=10^{-4}$ the two can be far apart.

Relation to Prior Work

Two papers motivate the gap. Adversarial Machine Learning in Network Intrusion Detection Systems shows that functional evasion is easy when only "mutable" features are perturbed: packet sizes, inter-arrival times, durations and window parameters, as opposed to fixed ones like protocol and ports. On UNSW-NB15, genetic-algorithm evasion rose from 5% to 73.71%, and tree models went from near 0% to 100% evasion. That paper is purely offensive. It never trains a defence or measures false alarms. We reuse its feature split (29 mutable features on NSL-KDD, 23 on UNSW-NB15) and its attack generators.

SA3C-ID is the closest defence. It trains a defender against a reinforcement-learning attacker using class-weighted rewards, and reports 92.58% F1 with a 1.2% false-alarm rate on NSL-KDD. Its attacker's actions carry no protocol guarantees, and its authors say U2R detection remains unsatisfactory. It also never ties perturbation strength to benign false alarms. A 1.2% rate is two orders of magnitude above the operating points we target.

Two papers supply method ingredients and limit the design. Confusion-Geometry Rebalancing for Long-Tailed Adversarial Training (CGRm) shows that long-tailed adversarial training suffers from both loss domination by large classes and boundary bias. Under attack, tail samples are pushed into specific nearby head classes. CGRm corrects this with confusion-graph margins and lifts tail robust accuracy on CIFAR10-LT from 13.19% to 27.93%. It was tested only on images at ratios of 10:1 and 50:1, and its confusion statistics become degenerate when a class has a few dozen samples. Our threshold-anchored loss needs no per-pair confusion counts.

Pixel Reweighted Adversarial Training (PART) shows that spending the perturbation budget unevenly recovers 1.2% to 3.18% clean accuracy on CIFAR-10. Our mutable-feature constraint set is the tabular, threat-derived analogue of that unevenness. It comes from protocol semantics rather than saliency maps, which do not exist for flow vectors.

On the imbalance side, two papers rebalance training data for ordinary (non-adversarial) training. S2CGAN-IDS uses tiered synthesis: generative models for moderately rare classes and nearest-neighbour interpolation for the rarest. It takes Heartbleed recall on CICIDS2017 from 0% to 100%. Detecting Network Intrusions Using Signal Processing with Query-Based Sampling Filter curates boundary pairs and reaches 0.996 benign precision on KDD99. Neither tests robustness.

Simplifying Neural Network Training Under Class Imbalance is a counterweight. It finds that small batches, sharpness-aware minimisation applied asymmetrically to minority classes, and class-conditional label smoothing beat specialised imbalance losses, and that a tuned multilayer perceptron beats XGBoost on tabular data. We adopt that recipe as the base trainer so that any gain we report is not just better imbalance hygiene.

Research Program

The work has three parts.

A functional threat set $\Delta(x)$ for flow features. It is built on the published mutable/immutable split and adds two kinds of constraints that keep perturbed flows physically consistent. The first is one-directional bounds where the semantics force them: an attacker can pad bytes or add delay, but cannot shrink a payload it needs. The second is consistency between derived features, such as mean packet size equalling bytes divided by packets. Every inner-loop step projects back onto this set. We will release the set as code so attacks and defences share one definition.

The training procedure. It implements the objective above with multi-step projected gradient ascent on the mutable coordinates. Hinge loss sits against the running benign quantile, and benign flows are left unperturbed. Ultra-rare classes are handled by upweighting them inside the attack term. CGRm-style confusion feedback is deliberately excluded, because with near-zero support it adds noise.

A controlled analysis of the trade-off. We subsample attack classes to fix imbalance ratios between 10:1 and 10,000:1 and measure how clean recall at fixed false-alarm rate degrades under each training scheme. This directly tests whether the imbalance-dependent part of the cost disappears when the threshold is anchored to benign quantiles. We also check whether the linear-model prediction survives unequal class covariances.

Baselines

Do nothing. The naive reference is the undefended detector: the tuned multilayer perceptron with the imbalance recipe of Simplifying Neural Network Training Under Class Imbalance, with its threshold set post hoc to each target false-alarm rate. An untuned random forest is reported alongside for continuity with prior intrusion results. The only reported real-world figure is ZYELL-NCTU NetTraffic-1.0's random-forest F1 of 0.3635. The undefended detector anchors how much clean recall at fixed false-alarm rate any defence gives up.

Symmetric constrained adversarial training. This is the strongest competitor on our specific question, and the one a careful practitioner would actually build. It is standard projected-gradient adversarial training that perturbs both classes within the same mutable-feature set $\Delta(x)$, uses argmax cross-entropy, applies the same imbalance recipe, and sets its threshold post hoc. It isolates exactly what we claim is wrong with ported vision practice.

The nearest published system. SA3C-ID is that system. Its reported NSL-KDD numbers are 92.58% F1 and a 1.2% normal-traffic false-alarm rate. We will reimplement it where possible and evaluate it under our attacks and operating points. If reimplementation proves unfaithful, we report its published figures only as clean context. No robust-recall-at-fixed-false-alarm figure is reported for it or for any adversarially trained intrusion detector: no reported result found on Lacuna for that metric.

A tabular port of CGRm. This variant uses class weights and confusion-pair margins on top of symmetric adversarial training. It tests whether long-tailed adversarial-training machinery already solves the problem. Its vision result, 27.93% tail robust accuracy on CIFAR10-LT, is not comparable across domains and will not be used as a target.

Oversampling plus symmetric training. Nearest-neighbour oversampling of the rarest classes in the style of S2CGAN-IDS, combined with symmetric adversarial training.

Ablations. Two ablations isolate our ingredients. "Attacker-only, argmax" perturbs only attack flows but attacks the 0.5 boundary. "Symmetric, threshold-aware" perturbs both classes against the quantile threshold. Together with the full method, these separate the one-sided threat model from threshold-aware inner maximisation.

Fair comparison. Every method uses the same splits, the same base network, the same constraint set at evaluation, the same number of inner attack steps, and an equal hyperparameter search budget. That budget is a fixed number of trials, chosen on a validation split by robust recall at the target false-alarm rate.

What counts as winning. Beating the symmetric baseline means higher worst-case attack recall at false-alarm rates of $10^{-3}$ and $10^{-4}$ with no more than one point lower clean recall at the same rate, averaged over seeds with non-overlapping confidence intervals. Separately, the full method should show a flatter clean-recall degradation curve across the imbalance sweep.

Why not just…?

Why not just run symmetric adversarial training and move the threshold afterwards? This is the real threat, and the experiment is designed around it. Hu et al.'s decomposition implies that a post-hoc threshold can repair the placement error caused by norm shrinkage, but not a boundary whose direction has tilted. Symmetric training also deliberately builds a margin around benign points, which can only push the boundary into attack territory. Still, if post-hoc thresholding of symmetric training matches our method at $10^{-4}$, the honest conclusion is that only the evaluation protocol needed fixing. We would report that as the finding. The "symmetric, threshold-aware" and "attacker-only, argmax" ablations show which ingredient carries any gain.

Why not just reweight classes, or port CGRm? Reweighting addresses which classes dominate the loss but not where perturbed tail samples land. CGRm itself shows this on images. CGRm's correction depends on per-pair confusion counts, which become degenerate when a class like U2R has a few dozen samples. The CGRm-port baseline under the imbalance sweep shows where it breaks down. If it holds up at extreme ratios, the threshold-anchored loss offers less than claimed.

Why not just oversample the rare attacks and skip adversarial training? Oversampling fixes recall on clean data. S2CGAN-IDS reports Heartbleed going from 0% to 100%. But it leaves the boundary as exposed as the undefended models in Adversarial Machine Learning in Network Intrusion Detection Systems. Interpolated synthetic flows may also violate the functional constraints, which could make adversarial training on them actively harmful. We test oversampling alone and oversampling with symmetric training under the same attacks.

Why not just use tree ensembles? They are the standard intrusion-detection tool, but on these very datasets they were the most fragile models tested, going from near 0% to 100% evasion. We report them as a clean reference and do not try to harden them.

Proposed Experiments

Datasets and splits. - NSL-KDD and UNSW-NB15 are the primary datasets, because published mutable-feature lists exist for them. - CICIDS2017 adds ultra-rare classes such as Heartbleed, with the functional constraint set derived the same way. - ZYELL-NCTU NetTraffic-1.0 tests whether the robustness gains survive its built-in drift between training and test periods. - All splits are stratified, five seeds each.

Attacks. Evaluation uses the genetic, particle-swarm and generative attacks from the mutable-feature study, plus constrained projected gradient ascent. It also includes an adaptive attack that knows the defence and targets the deployed threshold directly. A defence that only beats the argmax attack counts as failed.

Metrics. - Clean recall at benign false-alarm rates of $10^{-2}$, $10^{-3}$ and $10^{-4}$. - Worst-case attack recall at the same rates, per attack class and per family. - The trade-off cost: clean recall lost relative to the undefended detector at each rate.

First decisive check. Across the controlled imbalance sweep, does the trade-off cost of symmetric training grow with the ratio while ours stays roughly flat? A flat curve for both refutes the norm-shrinkage story in this domain. A rising curve for both refutes our remedy.

Second decisive check. On U2R and Heartbleed, does attacker-only threshold-aware training keep useful robust recall at $10^{-4}$, where SA3C-ID's authors report U2R already weak on clean data?

Third decisive check. Do perturbed training flows pass the functional constraint checks, so that measured robustness reflects realizable attacks rather than impossible ones?