Credit-Targeted Robust Training for Delayed-Reward Agents

Reinforcement learning agents can be broken by small, carefully chosen changes to what they observe. Defending them is expensive, because standard defenses search for the worst-case perturbation at every visited state, and that search costs many extra forward and backward passes per state. It also hurts performance: an agent trained to distrust every observation plays worse when nobody is attacking. This proposal starts from a simple observation. When reward arrives late, most states barely affect the outcome, and a few pivotal ones decide it. An agent that has to learn from delayed reward already trains a model of which past states earned the eventual payoff. We propose reusing that model to decide where to spend the defense. The expensive worst-case search and the robustness penalty are concentrated on the high-credit states and left thin elsewhere. If this works, it should cut the cost of robust training and also reduce the clean-performance penalty, since low-stakes states are no longer made pessimistic for no gain.

We test the idea in agents built from multi-compartment spiking neurons. These are neurons split into an input-driven compartment and a separate compartment that carries memory or context, and they recently reached 90.4% of a conventional recurrent world model's performance on 19 DeepMind Control tasks (Spiking world model with multicompartment neurons for model-based reinforcement learning). No work in the file has examined these agents under adversarial observations. They are also a setting where the defense's per-state cost is especially heavy: gradients must pass through spiking dynamics over time, and the bound-propagation shortcut used for ordinary networks is not readily available. The compartment split supplies a second, cheaper lever. We can restrict robustness updates to the weights that read observations and leave the memory-carrying compartment untouched.

Core Claim

In delayed-reward control, vulnerability to observation attacks concentrates on high-credit states. Weighting both the attack search budget and the robustness penalty by a learned per-state credit estimate should match uniform robust training on worst-case return. It should do so with a small fraction of the inner-loop compute and a smaller loss of clean return. We make a second, separable claim. In a two-compartment neuron, confining the robustness update to the input-facing compartment preserves clean behavior better than letting it touch the recurrent memory pathway.

Concretely, let $s_t$ be the observed state at step $t$ and $\pi$ the policy. Let $B(s_t)$ be the allowed perturbation set, $\kappa$ a penalty strength, and $w_t \ge 0$ a normalized credit weight produced by a return-decomposition model. We train with

$$\mathcal{L} = \mathcal{L}_{\text{RL}} + \kappa \sum_t w_t \max_{\delta \in B(s_t)} D_{\mathrm{KL}}\big(\pi(\cdot\mid s_t)\,\|\,\pi(\cdot\mid s_t+\delta)\big),$$

The inner maximization at step $t$ gets $K_t \propto w_t$ gradient steps under a fixed total budget $\sum_t K_t = K_{\text{total}}$. When $K_t = 0$, that state receives no attack search and no penalty. Setting all $w_t$ equal and $K_t$ constant gives back the uniform state-adversarial regularizer.

Relation to Prior Work

The robustness penalty comes from Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. That paper models observation attacks as a game in which the adversary perturbs what the agent sees but not the true dynamics, and it regularizes the policy uniformly across states. It avoids the expensive inner search by using convex relaxation bounds, which, as the file notes, still cost substantially more memory and compute than ordinary backpropagation. We keep its penalty but make it non-uniform. We use sampled gradient search, because relaxation bounds are not established for spiking multi-compartment dynamics. Its finding that the regularizer improved clean scores on Humanoid is a caution for us: in some tasks the uniform penalty acts as a helpful stabilizer, and thinning it could remove that benefit.

The motivation that vulnerability is temporally concentrated comes from Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning. That survey reports strategically timed attacks that drive performance to near zero while perturbing only 25% of frames. If attackers only need a few moments, defenders plausibly only need to harden a few moments too. That inference is exactly what our adaptive-attack experiments must stress.

The credit weights come from the delayed-reward literature. Learning Long-Term Reward Redistribution via Randomized Return Decomposition learns per-step rewards from subsequences of episodes whose only feedback is a final return. Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning produces explicit per-step importance weights with an attention mechanism and handles non-additive reward aggregation. Hindsight Credit Assignment scores actions by how much more likely they become given the realized outcome. These methods were built to improve learning, not defense. Their open problem is that they assume the recorded states are true, as noted in A Survey of Temporal Credit Assignment in Deep Reinforcement Learning, and we use them in the regime where that assumption is violated. We therefore recompute credit on attacked rollouts, not only clean ones.

The compartment split draws on two sources. TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling uses a non-leaky dendritic memory compartment beside a spiking soma to carry information across long sequences. The multi-compartment world-model agent above scales this to control. Neither evaluates attacks. The idea of restricting a defense to a small subset of parameters on a frozen backbone is borrowed from Robustness Tokens: Towards Adversarial Robustness of Transformers. There, training only about ten prepended tokens on a frozen DINOv2 recovered white-box ImageNet accuracy from 0.3% to 63.1% while clean accuracy moved only from 85.6% to 85.2%. Our analogue is to train only the observation-facing compartment's weights.

The closest robustness-plus-dendrites work is CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies. It adds cerebellar-style sparse expansion and fixed dendritic modulation to a value-based agent and tests it only against random Gaussian observation noise and sticky actions. It studies neither worst-case attacks, nor delayed reward, nor training cost.

Research Program

We build three agent families on shared code. The first is a conventional recurrent-world-model agent. The second is a point-neuron spiking agent. The third is a two-compartment spiking agent in which each neuron's input compartment receives the observation encoding and a second compartment receives the recurrent memory state. The file does not give the exact neuron equations of the published multi-compartment world model, so we define this split explicitly and report its clean performance against the published 90.4% figure as a sanity check.

Each family is trained under two reward conditions. One uses standard dense rewards. The other uses episodic delayed-reward versions in which the whole return is revealed only at episode end, following the protocol of the return-decomposition work. In the delayed condition a credit model is needed for learning anyway, so its weights $w_t$ are free for our purposes. In the dense condition we train one separately and count its cost.

The analysis has three parts. First, before any defense, we measure whether vulnerability actually concentrates. We attack one state at a time in held-out trajectories and correlate the resulting drop in return with $w_t$, with the policy's action gap, and with value magnitude. If credit weights predict vulnerability no better than the action gap, the credit machinery is unnecessary, and we would redirect the project toward gap-targeted training. Second, we train with the weighted objective above across total inner-loop budgets, and trace curves of worst-case return against inner-loop gradient evaluations and against clean return. Third, we compare restricting robustness gradients to the input compartment against applying them everywhere, in both the two-compartment agent and a point-neuron agent where the analogous restriction is to the first layer.

Baselines

The naive reference is the undefended agent of each family, trained with the same data and steps. It fixes clean return and shows how far performance collapses under attack. For scale, the state-adversarial paper reports PPO on Hopper falling from about 3167 to about 733 under strong attacks.

The strongest existing method is the uniform state-adversarial regularizer from Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. Its reported numbers are about 1224 on Hopper under strong attack for its PPO variant and a perfect +21 on Pong for its DQN variant, where undefended agents score −21. We run it in two forms. One uses its convex-relaxation bounds, on the conventional agent only. The other uses gradient-search inner maximization with a constant $K_t$, on all agents. We give it the same environment steps, the same total inner-loop budget swept over the same grid, and the same hyperparameter search trials for $\kappa$ and the attack step size. Beating it means matching its worst-case return within seed noise at a quarter or less of its inner-loop gradient evaluations, or matching its compute with higher clean return, on most tasks. There is no reported result found on Lacuna for any multi-compartment or spiking agent under adversarial observation attacks, so for those agents the uniform regularizer we train ourselves is the reference.

A parameter-restricted baseline follows Robustness Tokens: Towards Adversarial Robustness of Transformers. We freeze a cleanly trained agent and adversarially train a small input adapter under the uniform penalty. This separates the effect of restricting which parameters move from the effect of targeting which states are defended.

Two ablations isolate our ingredients. The first keeps the weighted objective but replaces $w_t$ with a random allocation at the same sparsity. The second keeps credit targeting but lets robustness gradients reach the memory compartment. This tests whether compartment localization contributes anything beyond what first-layer restriction gives a point-neuron network.

Why not just…?

Why not just subsample states at random for the inner search? This cuts compute by exactly the same factor, and if vulnerability were spread evenly it would perform as well. It is the most dangerous alternative and is one of our ablations. The timed-attack evidence in Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning suggests vulnerability is uneven, but that is evidence about attacks, not defenses. If random allocation at equal sparsity matches credit allocation, the targeting claim fails and we report it as such.

Why not just use the action gap or the value estimate to pick states? These heuristics need no extra model. Timed attacks typically choose moments where the policy strongly prefers one action. In dense-reward tasks they may well match learned credit, and the vulnerability-correlation study measures this directly. Our expectation that credit wins rests on the delayed-reward setting. There, value estimates near the pivotal early states are poorly learned, because standard temporal-difference learning favors recent actions, as documented in A Survey of Temporal Credit Assignment in Deep Reinforcement Learning. If the gap heuristic ties in the delayed condition too, we adopt it and drop the credit model.

Why not just train a conventional recurrent agent and ignore spiking neurons? For the targeting claim, the architecture should not matter, and we test it on the conventional agent first. The spiking multi-compartment agents matter for two reasons. They are the setting where no robust-training result exists and where per-state defense cost is highest. They also offer a built-in memory-versus-input parameter split for the localization claim. If localization gives no benefit over first-layer restriction in a point-neuron network, we will say the compartment structure adds nothing for robustness.

Why not just freeze the agent and train an input adapter? This is cheap and worked well for vision transformers. However, those results are for single images without closed-loop feedback. In control, an unrepaired perturbation at a pivotal state propagates through the agent's memory. The adapter baseline measures whether that matters.

Proposed Experiments

Environments are MuJoCo Hopper, Walker2d, HalfCheetah and Ant, in dense and episodic-delayed forms, as used for return decomposition and state-adversarial training. We add Atari Pong for the discrete-action case and a subset of the DeepMind Control tasks used by the multi-compartment world model. Attacks include untargeted gradient attacks at every step and strategically timed attacks restricted to 25% of frames. The decisive attack is adaptive. The attacker knows the credit model and deliberately spends its budget on low-credit states, including chains of small perturbations that steer the agent toward bad regions without touching pivotal states. It also sees credit recomputed on attacked rollouts.

Metrics are clean return, worst-case return across attacks, the adversarial regret (clean minus attacked return), inner-loop gradient evaluations, and wall-clock GPU hours including the credit model's cost. We use at least five seeds, with confidence intervals on all curves.

Three results would decide the claims. First, credit-weighted training should dominate random allocation and uniform training on the compute-versus-worst-case-return curve under the adaptive attacker in the delayed-reward condition. If it wins only against non-adaptive attacks, the method is a false economy. Second, clean return should be at least as high as under uniform training on every task except possibly Humanoid-like tasks, where the uniform penalty is known to help. Third, input-compartment localization should reduce clean-return loss relative to unrestricted updates by more than first-layer restriction does in the point-neuron agent. Failing the first check kills the proposal. Failing only the third reduces it to a result about delayed-reward robust training in which the compartment structure is incidental.