Gain-Spike Signatures: Detecting Backdoored Control Policies via Local Action-Gain Monitoring

A backdoored neural network behaves normally on clean inputs and switches to malicious behavior when a small, hidden trigger appears. In image classification this switch is invisible to standard validation; in continuous control it is, in principle, loud. A trigger is a tiny perturbation $\delta$ of the sensor state that flips the policy's output from a benign command $u_{ben}$ to a malicious one $u_{mal}$, so the policy's local input-to-output gain — its empirical Lipschitz constant — must spike to roughly $\|u_{mal} - u_{ben}\| / \|\delta\|$ exactly at the trigger. Meanwhile, an entire line of safety-critical control research exists to keep that same gain bounded, because controllers whose output jumps discontinuously with the state chatter, destroy actuators, and void the existence and uniqueness guarantees of the closed-loop dynamics (Tabuada et al., Cortes et al.).

This proposal claims that these two facts belong together. Trigger activation in a continuous-control policy is, at its core, a regularity anomaly, and regularity is something we can measure cheaply at runtime, without access to model weights, without clean training data, and without a classifier's entropy to lean on. The research program builds a runtime monitor that probes the policy with small state perturbations and flags gain spikes; pairs it with a Lipschitz-guaranteed safety filter so that even undetected triggers cannot produce physically destructive commands; and then attacks both components with a deliberately Lipschitz-stealthy backdoor to map the real limits of the defense. The closest existing work stops short on all three sides: PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning crafts backdoors in constrained RL but scores them only by cumulative cost, never looking at the actuator signal; Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets poisons offline RL datasets but measures reward collapse, not smoothness; and the regularity theory of Reformulations of Quadratic Programs for Lipschitz Continuity and Continuity and Boundedness of Minimum-Norm CBF-Safe Controllers assumes the filtered controller is benign. Nobody has asked what a Trojan does to a safety filter, or what a safety filter does to a Trojan.

Core Claim

Let $\pi$ be a continuous-control policy and let a trigger be a sparse, small perturbation $\delta$ of the observed state. The detector estimates the local action gain at the current state $x$ by querying the policy at a handful of nearby states:

$$\hat{L}(x) \;=\; \max_{v \in \mathcal{V}} \frac{\|\pi(x + \epsilon v) - \pi(x)\|}{\epsilon},$$

where $\mathcal{V}$ is a small fixed set of probe directions (including directions aligned with plausible sensor channels) and $\epsilon$ is small. The claim is threefold. First, on clean states a well-trained continuous-control policy has a moderate, roughly stationary $\hat{L}$, whereas at a trigger boundary the gain spikes by orders of magnitude, because the benign and malicious action modes are separated by a large action distance across a thin input region. Second, this signal supports a runtime detector with a probe budget of a few extra forward passes, in contrast to classifier-era defenses that need class-probability entropy or weight access. Third — and this is where the safety-filter literature becomes load-bearing rather than decorative — detection alone does not bound actuator damage, so the monitor is paired with the closed-form, globally Lipschitz safety filter of Reformulations of Quadratic Programs for Lipschitz Continuity, which confines the commanded action to a ball of radius $r(x)$ around a known Lipschitz nominal policy $\pi_f(x)$. The two components squeeze the attacker from opposite ends: a step-like trigger trips the gain monitor; a gradual trigger avoids the monitor but gives the bounded, Lipschitz filter time and authority to keep the trajectory and the actuator signal viable.

Research Program

The first workstream is characterization rather than defense. We reproduce two published attack families on continuous-control benchmarks: the BAFFLE offline-RL poisoning pipeline of Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets on D4RL Hopper, Half-Cheetah, and Walker2d, and the backdoored constrained MDP construction of PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning on Safety-Gymnasium PointGoal, CarGoal, and AntGoal, using environments and algorithm implementations from Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark. For each backdoored policy we measure $\hat{L}$ along clean rollouts and along rollouts crossing the trigger manifold, and test the separation hypothesis. This is the empirical foundation everything else rests on; if the gain distributions overlap, the core claim is wrong and we say so.

The second workstream studies the interaction the regularity literature has never examined: what happens when a backdoored policy sits upstream of an optimization-based safety filter. We wrap the backdoored desired control $u_{des}(x) = \pi(x)$ in two filters — the standard minimum-norm CBF quadratic program, and the SOCP reformulation of Reformulations of Quadratic Programs for Lipschitz Continuity — and measure chattering (root-mean-square of step-to-step action change), peak commanded action, and constraint violations. A specific worry from Continuity and Boundedness of Minimum-Norm CBF-Safe Controllers guides this: that paper proved minimum-norm CBF controllers can become unbounded near a singular boundary set $Z$ where the control authority $L_g h(x)$ vanishes, showing peaks of $10^3$ versus a bounded maximum of 0.9 on their double-integrator benchmark. We will test whether a trigger can be engineered to steer the state toward $Z$, turning a provably safe architecture into a physically unimplementable one.

The third workstream is the adaptive attack that makes this a proposal rather than an observation: a Lipschitz-stealthy backdoor. The attacker replaces the step switch with a ramp — the malicious mode is entered gradually, over a time constant $\tau$, or across a smeared region of state space — so the instantaneous gain stays below any detector threshold. We formalize the expected tradeoff: ramping divides the local gain roughly by $\tau$ but multiplies the time before full malicious authority is reached, during which the Lipschitz filter, the state monitor, and the operator can respond. The decisive output is a measured evasion-versus-efficacy frontier: detector evasion rate and time-to-constraint-violation as functions of $\tau$, against both the monitor-only and the monitor-plus-SOCP defenses. This tells the community not just that regularity monitoring helps, but exactly how far an attacker can push back.

Baselines

The naive reference point is doing nothing: deploy the backdoored policy with no defense. This is not a strawman, because doing nothing is the default deployment mode, and its cost is quantified in the file: BAFFLE reports that under trigger the poisoned offline-RL policies' performance plummets by 63.2% in Hopper (and 47.4% in Carla) while clean performance drops only about 3.4% at a 10% poisoning rate, meaning an operator who does nothing ships a policy that looks fine and fails catastrophically on cue (Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets).

The strongest existing runtime defense we will compare against is the subspace sanitization wrapper of Provable Defense against Backdoor Policies in Reinforcement Learning, which estimates a clean state subspace $E$ by singular value decomposition on roughly 20 clean calibration rollouts and runs the policy on the projected observation; it recovered nearly 100% of clean performance on Atari while scores plummeted to near zero under the unfiltered attack. It is the strongest competitor because it is the only published defense with a formal guarantee and demonstrated near-complete recovery in reinforcement learning, so our claim must beat it on its weakest flank: it has never been run in continuous control, and its guarantee assumes the trigger lives in dimensions orthogonal to the task subspace. We run it on identical poisoned Safety-Gymnasium and D4RL setups with the same calibration data and tuning budget as our monitor, and also with a deliberately non-orthogonal trigger embedded in task-relevant sensor channels (lidar-like distance bins and velocities). Beating it means higher detection AUROC or lower attack success at equal clean-performance retention, and — critically — robustness on the task-embedded trigger where its projection must either pass the trigger through or destroy task signal.

A second published baseline is a continuous-action adaptation of STRIP (STRIP: A Defence Against Trojan Attacks on Deep Neural Networks), which perturbs the input and watches prediction entropy; it reported 0% false acceptance and 0% false rejection rates on CIFAR10 and GTSRB in most cases, under 1% FAR at 1% FRR in hard cases, at a 1.32x inference latency cost. Since entropy is undefined for continuous actions, we adapt it by replacing entropy with the variance of the perturbed action outputs — an honest, strongest-reasonable reading of the method in our setting, run with the same probe budget as our monitor. Finally, the key ablation isolates each ingredient: the gain monitor alone (no Lipschitz filter) versus the SOCP filter of Reformulations of Quadratic Programs for Lipschitz Continuity alone (no monitor). The monitor should catch step triggers but not bound damage; the filter should bound damage but not flag anything; only the combination should survive the ramped adaptive attack. If the filter alone matches the combination, the monitor is decoration and the proposal should shrink to a filter study.

Why not just…?

Why not just fine-tune the suspect policy on clean data? Because in this regime it demonstrably fails: BAFFLE tested exactly this and clean fine-tuning recovered less than 3% of performance in offline RL, with Neural Cleanse and Activation Clustering also failing to detect the poisoned transitions, because continuous trajectories let poisoned activations blend into normal distributions (Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets). Fine-pruning fares better in classification — Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks reduced attack success from 100% to 0% in face recognition — but its own results show degradation on harder tasks (roughly 99% down to only about 30% on traffic signs), and it needs internal weights and clean labeled data, which a downstream operator running a policy as a black-box executable does not have. We include fine-pruning where weight access exists, to delimit when retraining suffices.

Why not just wrap the policy in a standard CBF quadratic program? This is the most serious alternative, and we test it directly rather than argue around it. The QP guarantees the state stays in the safe set, but not that the command is implementable: standard CBF-QPs are provably non-Lipschitz when constraint qualifications break down, producing chattering that damages hardware (Tabuada et al., Reformulations of Quadratic Programs for Lipschitz Continuity), and minimum-norm CBF controllers can blow up to $10^3$ near the singular set $Z$ where a triggered attacker can deliberately steer (Continuity and Boundedness of Minimum-Norm CBF-Safe Controllers). Our experiments include the plain QP filter as a condition, and the honest outcome we anticipate is: the QP prevents state constraint violations from step triggers but passes chattering and blow-up through to the actuator, which the gain monitor plus SOCP filter does not. If the plain QP in fact stays smooth on these benchmarks, the actuator-damage half of our motivation collapses and the proposal reduces to the detection story.

Why not just use STRIP? In spirit, our monitor is a continuous-action relative of STRIP, and we treat it as a genuine threat rather than pretend otherwise. The differences are concrete: STRIP's signal is Shannon entropy over class probabilities, which does not exist for an action vector in $\mathbb{R}^m$; its 1.32x latency overhead from blending many perturbed copies is a real cost in loops running at 100 Hz to 1 kHz, whereas our gain probe uses a handful of directional differences; and, most importantly, STRIP only detects — it does nothing to bound what the actuator receives in the episodes where detection fails or fires late. Our head-to-head against the STRIP-adapted baseline with matched probe budgets is designed to settle whether the gain signal specifically (rather than perturbation per se) carries the information, and the ramped-attack experiments settle whether detection alone is ever enough.

Why not just project observations onto a clean subspace? Because the defense of Provable Defense against Backdoor Policies in Reinforcement Learning rests on the trigger being orthogonal to the task subspace, and an attacker who reads the paper simply embeds the trigger in task-relevant dimensions — a perturbed lidar bin or velocity channel in Safety-Gymnasium — where projection cannot remove the trigger without removing the task. It has also only ever been validated on discrete-action Atari (Boxing-Ram, Breakout), so its effect on the smoothness of continuous torque commands is unknown and possibly negative, since a projection can itself introduce discontinuous state transitions. We run it as our strongest baseline precisely to test both failure modes empirically.

Proposed Experiments and Decisive Checks

Experiment one runs the characterization: BAFFLE-poisoned policies (10% poisoning rate, matching the published setup) on D4RL Hopper, Half-Cheetah, and Walker2d, and PNAct-style backdoors on Safety-Gymnasium PointGoal, CarGoal, and AntGoal across PPO-Lag and RCPO. We report the distribution of $\hat{L}$ on clean versus trigger-adjacent states, the detection AUROC, and the false-positive rate on clean rollouts. The decisive check is separation: AUROC near chance would falsify the core claim. Experiment two is the filter-interaction study, including the double-integrator boundary benchmark from Continuity and Boundedness of Minimum-Norm CBF-Safe Controllers to test trigger-steering into the singular set $Z$, comparing minimum-norm QP against the SOCP filter on chatter, peak command, and Lipschitz constant of the closed-loop command signal. Experiment three is the adaptive Lipschitz-stealthy attack: sweep the ramp constant $\tau$, report detector evasion and time-to-constraint-violation against monitor-only, filter-only, and combined defenses. Experiment four is the full comparison against all baselines on identical poisoned environments and tuning budgets, with metrics fixed in advance: attack success rate, clean performance retention, peak and RMS actuator command, latency overhead per control step, and clean-data requirements. No reported result found on Lacuna for any existing defense on these continuous-control poisoning benchmarks, which is itself the point: the numbers we produce here would be the first.