Pruning What the Policy Should Ignore: Token Dropping Against Visual Overfitting

Agents that learn to act from pixels often break when something irrelevant changes, such as the background color, a moving distractor, or the lighting. The game rules and the physics stay the same, yet the agent fails, because during training it latched onto whatever pixels happened to correlate with reward. Vision Transformers (image models that cut a frame into patches, or "tokens," and let every patch attend to every other) already have a well-developed way to throw away patches layer by layer: progressive token pruning. So far it has been used only to save compute. We propose using it as a hard information bottleneck. If a background patch is physically removed from the sequence before the policy's deeper layers see it, then a change to that background cannot reach the decision. The policy is protected only if the part of the network that chooses which patches to drop does not itself overfit to the training backgrounds.

The idea in two sentences: train a small patch scorer whose targets come from the agent's own behavior (how much the action distribution and value estimate change when a patch is hidden), and train that scorer on heavily augmented frames while the policy sees clean frames through the pruned sequence. The result should be a compact, specialized visual policy that is cheaper to run and that fails less under background and distractor shifts. It should not help under global photometric shifts, and we state this in advance as a prediction.

Core Claim

Where and how the pruning decision is learned determines whether token pruning reduces visual overfitting. We claim three things. First, pruning with a scorer supervised by behavioral counterfactuals and trained on augmented views gives larger out-of-distribution gains on background and distractor shifts than random token dropout, raw-attention Top-K pruning, or input augmentation alone, at a matched fraction of tokens kept. Second, the gain is specific to the kind of shift. It should be large where the nuisance occupies separable patches (background, distractors) and near zero where it touches every patch (camera filters), because a patch-level bottleneck cannot remove a perturbation spread across all patches. Third, the same mechanism reduces the policy encoder's attention FLOPs, so specialization to a control domain costs less compute rather than more.

The second claim is what makes the proposal falsifiable. If gains appear uniformly across all shift types, the mechanism is not doing what we say. That would suggest a generic regularization effect, which random dropout should then match.

Relation to Prior Work

Progressive pruning supplies the method. DynamicViT inserts small MLP predictors at blocks 4, 7 and 10 of a ViT. It trains them with Gumbel-Softmax, a relaxation that makes discrete keep/drop choices differentiable, and uses attention masking. On DeiT-S it removes 66% of tokens for a 0.5-point ImageNet drop (79.8% to 79.3%), and visibly discards sky and branches. Adaptive Token Sampling and PoWER-BERT score tokens by reusing attention, with no extra parameters. We reuse the three-stage schedule and the masking trick, but change the supervision signal and the objective from efficiency to robustness. None of these papers evaluates under distribution shift.

Which Tokens to Use? constrains the design. It finds that raw-attention Top-K matches or beats learned selectors in supervised vision. We therefore treat attention Top-K as a first-class baseline. If it matches our scorer under shift, we drop the learned scorer. We expect it to fail under shift for the reason Sparse Masked Attention Policies names: any selector trained on the training backgrounds can inherit their spurious correlations.

That paper is the closest existing work. It learns stochastic hard masks over pairwise attention edges on CNN features, with a hand-set sparsity target, and roughly doubles Bigfish returns on Procgen. It never removes tokens, so it saves no sequence length. Its selector is also trained on the same observations as the policy. Our proposal removes whole patches, supervises the selector with behavioral counterfactuals, and decouples the selector's inputs from the policy's.

Two other papers are near neighbors. Predicting What Matters (MWM) uses random token pruning only as a diagnostic stress test and relies on segmentation annotations. Learning to Focus (LeaF) is the conceptual precedent for pruning "confounding" tokens identified by gradient sensitivity. It works on text, and its confounders are found by comparing against a 70B teacher. We replace the teacher with the agent's own policy and value heads, which is what lets the method run inside RL without any external model.

Three lines of visual-overfitting work motivate the gap and supply one ingredient. RAD shows that crop and translation help but are "less effective on tasks requiring structural generalization." SODA and SADA show that strong augmentation destabilizes value learning unless it is confined to a side objective or to the actor. We borrow SADA's asymmetry and push it one step further: augmentation is applied only to the patch scorer, so the critic and actor never see augmented pixels. Deep Bisimulation for Control grounds representations in behavioral equivalence at a pairwise cost quadratic in batch size. Our per-patch counterfactual target is a cheaper, local version of that intuition. It asks whether a patch changes behavior, not whether two states behave alike.

Domain specialization is the application setting. AgriPerceiver specializes a compact model by squeezing 3,645 tokens into a fixed 128 latents and training only about 9.2% of its parameters. Its authors report that the fixed bottleneck hurt spatial localization. Our specialized policies keep a variable, content-dependent set of original patches, so they retain spatial position. Specializing Smaller Language Models warns that specialization costs out-of-domain ability. We test the visual analogue: a scorer specialized to one domain may prune task-critical patches when layout changes.

Research Program

The policy is a small ViT (DeiT-Tiny scale) trained with PPO. At blocks 4, 7 and 10 of 12, a light MLP scorer reads each patch feature plus a pooled global summary, as in DynamicViT, and outputs a keep probability. During training, dropped patches are masked out of attention. At inference they are physically removed, so the FLOP savings are real.

The scorer's target is behavioral. For a frame $x$ and patch $i$, let $x_{\setminus i}$ denote the frame's token sequence with patch $i$ masked. Define

$$ t_i(x) = \mathrm{KL}\big(\pi(\cdot \mid x)\,\|\,\pi(\cdot \mid x_{\setminus i})\big) + \lambda\,\big|V(x) - V(x_{\setminus i})\big|, $$

where $\pi$ is the current policy's action distribution, $V$ is the critic's value estimate, and $\lambda$ balances the two terms. Computing this separately for every patch is too expensive. Instead, on a small fraction of each batch we mask random subsets of patches and regress the resulting behavior change onto patch indicators, which amortizes the estimate across many masks.

The scorer $g$ is trained to rank patches by $t_i(x)$, but it is fed an augmented view $\tilde{x}$ (random convolution, color jitter, overlaid video frames). Its loss is a ranking loss between $g(\tilde{x})$ and $t(x)$. The targets always come from clean frames, and the policy and critic are updated only on clean frames passed through the pruned sequence. This separation is the answer to the "generalization paradox." The selector is pushed to be invariant to nuisance appearance, while value learning is never exposed to augmentation noise.

The keep ratio is set by the fraction of total behavioral score retained rather than by a fixed count. Cluttered frames therefore keep more patches and sparse frames keep fewer. For low-resolution Procgen and KAGE frames we also report fixed-ratio variants so that comparisons at matched keep fractions are exact.

The analysis has three parts. We measure which patches get dropped, both in training and under shift. On KAGE-Bench the renderer tells us which pixels belong to background and distractors, so we can compute the fraction of foreground patches wrongly dropped. We measure, per shift axis, the gap between in-distribution and out-of-distribution success. And we measure wall-clock throughput, which accounts for the batch irregularity that variable-length sequences introduce.

Baselines

The naive reference point is a dense ViT policy trained with PPO under the same budget, alongside the PPO-CNN that KAGE-Bench evaluates. For that CNN, KAGE-Bench reports an average success-rate drop of 86.8% under camera-filter shifts and 53.3% under background shifts. That paper reports no ViT or augmentation baseline, so we find no result on Lacuna for RAD or token pruning on KAGE-Bench. We run those ourselves.

The strongest existing competitor for robustness is input augmentation. We use RAD's random crop/translate. On Procgen, RAD reports a 55.8% relative gain over PPO on BigFish. We also run SADA-style asymmetric augmentation, because it is the most direct test of whether moving augmentation into the selector adds anything. Both are run on the identical dense ViT with the same environment steps, and each gets the same hyperparameter search budget: eight configurations per method, chosen on a held-out in-distribution validation split. Configurations are never chosen on the shifted test.

The strongest attention-sparsity competitor is Sparse Masked Attention Policies, which reports Bigfish 21.61 versus 11.32 for dense attention and roughly 0.8 versus 0.4 normalized return on Dodgeball. We reimplement it on our tokenization and also compare against its published Procgen numbers.

Ablations isolate each ingredient at matched keep fractions. Random token dropout isolates whether selection matters at all. Parameter-free attention Top-K isolates whether learning a scorer matters. A DynamicViT-style scorer trained only through the task loss isolates the behavioral target. Our scorer trained on clean rather than augmented views isolates the selector-invariance claim. Edge masking without token removal isolates physical dropping. We also add an oracle that prunes using the renderer's ground-truth foreground mask, which bounds what any patch selector could achieve.

To count as beating the baselines, the method must reduce the out-of-distribution success drop on KAGE background and distractor axes relative to the best of RAD and SADA, with non-overlapping confidence intervals over at least five seeds. It must also match their in-distribution success and use fewer encoder FLOPs.

Why not just…?

Why not just use data augmentation? Augmentation is the real threat, and on backgrounds it may already be enough. RAD itself concedes weakness on structural generalization, and SODA and SADA exist because augmented pixels harm value learning. Augmentation also leaves every background patch in the sequence. The experiment that settles this is the RAD/SADA-versus-ours comparison on each KAGE axis, together with the combination of augmentation plus pruning. If pruning adds nothing on top of SADA on the background axis, the robustness claim fails, and only the efficiency claim survives.

Why not just drop tokens at random during training? Random dropout is a known regularizer, and MWM's probe shows that robust models tolerate missing patches. If random dropout matched us, our scorer would be unnecessary. The prediction separating the two is the pattern across shift types. Random dropout should help all axes modestly, while behavioral pruning should help background and distractor shifts strongly and filter shifts not at all. The ground-truth foreground-drop rate also shows directly whether our selection differs from chance.

Why not just use attention Top-K, which Which Tokens to Use? found as good as learned selectors? Under the training distribution it may well be. Our hypothesis is that attention maps trained on fixed backgrounds shift when the backgrounds shift, as the Sparse Masked Attention paper argues. If Top-K matches our scorer under shift, we adopt it and drop the learned scorer.

Why not just mask the background with a segmentation model? On Frostbite, object masks greatly improve sample efficiency, but they cause "pixel neglect," and running segmentation at deployment adds latency and cascading errors (MWM). We include the renderer-mask oracle as an upper bound, not a competitor. The interesting quantity is how much of the oracle's gain an annotation-free selector recovers.

Why not just compress into a fixed latent set like a Perceiver? AgriPerceiver's fixed 128 latents lost spatial localization without added position embeddings, and a fixed budget cannot adapt to clutter. We include a Perceiver-bottleneck policy at matched latent count to test whether the difference matters for control.

Proposed Experiments and Decisive Checks

KAGE-Bench is the primary testbed. Its known shift axes and throughput of up to 33M steps per second on one GPU make many seeds and ablations affordable. We use all 34 train/evaluation pairs and report success rate, not return, since KAGE-Bench shows that returns can hide task failure. Procgen, trained on 200 levels and tested on the full distribution, tests structural generalization and allows comparison to Sparse Masked Attention and RAD. DeepMind Control with Kinetics video backgrounds tests continuous control. DBC reports "nearly identical to clean" performance there, with no single number given in the file. As a domain-specialization stretch, we fine-tune only the scorer and LoRA adapters of a pretrained ViT on LIBERO and test under background shift. There, MWM reports 27.5% success versus 3.8% for GE-ACT.

Three results would be decisive. First, the pattern of gains across shift axes: large on background and distractor shifts, near zero on filters. Second, the augmented-selector ablation: training the scorer on clean views should reproduce the overfitting that Sparse Masked Attention predicts. Third, a layout-shift test for a specialization tax: the scorer's wrongful foreground-drop rate should rise when layout changes, and if it does, the method's limits are located inside the selector.

If the shift pattern appears and random dropout does not reproduce it, token pruning earns a second role as a structural guard against visual overfitting. If not, we will have shown cleanly that its benefit in control is limited to saving compute.