Recycling Feasible Perturbations for Cheap Adversarial Training on Constrained Tabular Data

Adversarial training is a defense in which a model is repeatedly retrained on inputs that an attacker has nudged to cause errors. It is expensive because every training step must first search for a good nudge. On images, the search is the whole cost. On tabular data such as credit files, network flow records, or clinical measurements, there is a second cost. Every nudged record must also stay possible: no negative ages, no flows whose payload exceeds their total size, and derived columns that still match the raw columns they are computed from. Repairing records so they satisfy these domain rules means calling a solver, rerunning a feature pipeline, or propagating changes through a learned causal model. These repairs are slow, and current constrained defenses redo them from scratch at every attack step.

We propose to keep, for each training record, a persistent attack that is valid by construction. It is stored as an edit to the features an attacker can actually change, and the dependent features are recomputed from that edit. The stored attack is then refreshed with a single cheap gradient step per epoch instead of being rebuilt. Our central bet is that valid attacks of this kind carry over from one training epoch to the next better than unconstrained ones do, because they exploit structure in the data rather than quirks of the current weights. That would make recycling them both cheaper and safer in tabular domains than it is in vision. The bet is testable directly: if it holds, a model trained this way should match full multi-step constrained adversarial training in robustness at a fraction of the wall-clock cost, including the cost of the repair steps.

Core Claim

Two existing results point at the idea, and nobody has joined them. The first is ATTA, which reuses image perturbations across epochs because the weights change slowly. With one gradient step per epoch it matches PGD-40 robustness, reporting a 14.1× speedup on CIFAR-10. ATTA works only inside an $\ell_\infty$ or $\ell_2$ ball, where "project back into the ball" costs nothing. The second is LCSA. It perturbs only the parent variables of a learned causal graph and lets the changes ripple to child features. It finds that these structure-respecting attacks transfer from neural networks to tree ensembles such as LightGBM and XGBoost because they follow the data's structure. LCSA is purely an attack, and it is expensive per sample.

Our claim is that the same mechanism LCSA identifies for cross-model transfer also produces cross-epoch transfer during training. If so, ATTA-style recycling should not merely survive the move to constrained data; its gains should be larger there. ATTA saves only gradient passes. On constrained data, recycling also lets training skip most calls to the expensive feasibility repair.

FENCE reports weak and asymmetric transfer across architectures under its greedy family-update attack. That result constrains the proposal, because it suggests constrained attacks may be brittle across models. We treat cross-epoch carry-over as an empirical question we measure first, not an assumption.

Research Program

The method has three parts.

The first part is a valid-by-construction parameterization of the attack. We split the features into mutable inputs $u$, which the attacker can edit, and derived features $d = g(u)$. The derivation map $g$ covers explicit formulas such as ratios, sums, one-hot groups, and FENCE's dependency families. Where formulas are unavailable, $g$ is replaced either by a learned causal ripple map in the style of LCSA or by a profile estimator in the style of Melo et al.. The attack variable is an edit $\delta_u$, held within a domain cost budget of the kind used in cost- and utility-aware tabular robustness. Categorical edits are relaxed during the gradient step and rounded afterwards. Linear inequality rules are enforced by the differentiable constraint layer from Dyrmishi et al..

The second part is a persistent bank holding one edit per training record. Each epoch, the stored edit receives one gradient step against the current weights. A cheap validity check follows. The full solver-based repair runs only when the check fails, so repair becomes lazy rather than routine.

The third part is a restart policy. Each epoch, a small random fraction of entries is reinitialized from scratch. This guards against the bank going stale and against catastrophic overfitting, a known failure mode of single-step training in which the model suddenly loses robustness to multi-step attacks late in training.

The analysis component measures epoch-to-epoch carry-over directly. An edit computed at epoch $t$ is applied to the model at epoch $t+k$, and we record how often it still succeeds. We compare constrained and unconstrained edits under matched budgets. We also log the fraction of training time spent on repair, which tests the premise that repair rather than gradient computation dominates cost in constrained training.

Baselines

The naive reference point is standard training with no adversarial component. We evaluate it under the same attack suite to establish how much robustness any defense must recover.

The second baseline is unconstrained PGD adversarial training: a multi-step attack inside a feature-space ball, with no repair. It is the default practitioner choice. It is fair to include because the constrained threat set is a subset of a sufficiently large ball. We tune its ball radius on validation data, using the same grid budget as our method.

The strongest competitor is full multi-step constrained adversarial training, the approach FENCE evaluates for its own attack. Here every inner step is a fresh constrained gradient step followed by complete repair, using the same derivation map, constraint layer, and cost budget as our method. It is the right competitor because it is what our method claims to approximate. It shares our threat model, so it isolates the contribution of recycling alone. We run it with 10 and 20 inner steps, on the same splits, architectures, and number of epochs, with the same hyperparameter search budget.

No reported result for full constrained adversarial training on URL, LCLD, WiDS, CTU-13, or HELOC was found on Lacuna. FENCE states that adversarial training improves resilience but gives no figure in the file. The closest reported numbers come from different settings and cannot be compared directly. The cost-aware training paper reports a 40% reduction in attack success rate on HomeCredit. Melo et al. report preventing performance drops of approximately 30% under moderate attacks, with a clean-accuracy cost below 7%, on proprietary data. We will therefore produce the strongest baseline's numbers ourselves.

Our method beats this baseline if two conditions hold. Its robust accuracy under the strongest adaptive constrained attack must fall within one standard deviation across five seeds of the 20-step version. Its total wall-clock time, including repair, must be at least 5× lower.

Two ablations isolate the key ingredients. The first is "naive ATTA on tabular data." It accumulates unconstrained edits across epochs and projects only at the end, which tests whether the valid-by-construction parameterization matters. The second is "constrained single-step from fresh initialization." It uses our parameterization with no bank, which tests whether recycling matters.

Why not just…?

Why not just train on unconstrained PGD, since its perturbation ball contains every valid attack? On the Robustness of Domain Constraints finds up to 82% of PGD examples violate domain rules. TabAttackBench finds PGD produces the least plausible records. In malware, problem-space hardening was needed after defenses built in feature space were fully bypassed. The open question is whether those impossible points waste model capacity and raise the clean-accuracy cost. Our unconstrained baseline answers it on tabular transformers, which no paper in the file has done.

Why not just run constrained adversarial training with one or two fresh steps? This is the real threat. If tabular models are small enough that a few constrained steps already reach strong attacks, the bank adds nothing. The second ablation tests exactly this. If it matches our method's robustness under adaptive attacks, we will report that recycling is unnecessary and that the speedup comes entirely from the parameterization. That outcome would still be a useful negative result for the field.

Why not just train an offline generative attacker once and augment the training data? AdvDGMs reach 95% attack success rate on HELOC with TabTransformer. However, their attack success on WiDS falls below the clean model's own error rate. A fixed generator also cannot track a model whose weights are hardening. We include an AdvDGM-augmentation arm with matched compute to measure this directly.

Why not just reject invalid or anomalous inputs at inference? Valid attacks are exactly the ones that pass validation. LCSA reports only 19.4% Isolation Forest detection on URL, versus 61.4% for the next-best gradient attack. A filter therefore does not address the threat we target.

Proposed Experiments

We use the benchmarks where LCSA and the AdvDGM papers report results: URL, LCLD, WiDS, CTU-13, and HELOC. The models are an MLP, TabTransformer, and FT-Transformer.

Evaluation is deliberately harder than training. We attack with LCSA, CAPGD, FENCE, and AdvDGMs, and with adaptive attacks that know the defense. The metrics are clean accuracy, robust accuracy, attack success rate, the validity rate of training perturbations, and wall-clock time broken down into gradient computation and repair.

For transfer, we craft LCSA attacks on independently trained neural surrogates and apply them to each defended model. We also check whether a cheaply hardened neural model's bank of edits, used as augmentation for XGBoost and LightGBM, reduces LCSA's reported ability to transfer to trees.

Three checks decide the outcome. The first is the epoch carry-over measurement: if constrained edits do not carry over better than unconstrained ones, the central bet fails. The second is a late-training multi-step attack, which detects catastrophic overfitting. The third is the comparison with fresh constrained single-step training, which decides whether recycling earns its place.