Routing That Remembers: Amortizing Capsule Agreement Across Time to Reject Background Distractors in RL

Reinforcement learning agents that learn from pixels often memorize what the training levels look like instead of learning what matters for the task. When a background texture happens to correlate with reward, the image encoder picks it up, and the policy fails on new levels. Capsule networks offer a possible structural fix. These networks group neurons into small vectors, one per candidate object, and use an agreement-based "routing" step to decide which low-level parts belong to which object. Only features that vote consistently for the same object get bound together. Diffuse background texture should form no coherent object and so should receive little weight. The reason nobody runs capsules inside RL is cost. Routing-by-agreement is an iterative inner loop of three to five rounds per forward pass, and an RL agent makes tens of millions of forward passes.

Our proposal is to spread those routing rounds across environment timesteps instead of removing them. Consecutive frames in a game are nearly identical. The part-to-object assignments from the previous frame are therefore an excellent starting point, and a single refinement round per frame should track what full iterative routing would compute. This gives iterative routing's selectivity at the cost of one round. We also make a deliberately narrow, falsifiable prediction. Capsule routing should reject diffuse distractors such as backgrounds and video textures. It should not reject compact spurious objects such as an on-screen timer, because a timer is a coherent object that routing will happily bind.

Core Claim

A capsule encoder whose agreement logits are carried from frame to frame, with one routing round per step, will achieve two things. First, it will match three-round routing in how well it separates objects from background, at roughly single-pass cost. Second, on procedurally generated games, it will shrink the gap between training-level and test-level performance specifically when the shift is in background appearance. It will beat parameter-matched CNNs and single-pass attention routers there. Against compact spurious objects we expect no advantage. Confirming both halves would show what kind of observational overfitting part-whole routing addresses. That is more useful than a blanket "capsules generalize better."

Relation to Prior Work

The motivating evidence comes from three diagnoses of observational overfitting. Observational Overfitting in Reinforcement Learning splits observations into task features and level-specific distractors. It shows that overparameterization implicitly favors the task features, and it cures the Sonic timer problem only by blacking the timer out by hand. We replace manual masking with a learned grouping mechanism, and we use their timer setting as the case where we predict failure. Natural Environment Benchmarks for Reinforcement Learning shows that agents collapse when static backgrounds become natural video. It proposes no architectural remedy, and it is exactly the diffuse-distractor regime we target. Quantifying Generalization in Reinforcement Learning shows that CoinRun agents trained on 100 levels solve about 65% of test levels. It shows that dropout, L2, and Cutout narrow but do not close this gap. These regularizers act uniformly on the whole image, whereas routing acts per input.

The method comes from the capsule literature. Dynamic Routing Between Capsules introduced routing-by-agreement with an iterative loop. Three lines of work removed that loop for supervised vision: attention in a single pass (Attention Routing Between Capsules), inverted dot-product attention with layer normalization (Capsules with Inverted Dot-Product Attention Routing), and straight-through gated attention (STAR-Caps). A fourth approach replaced routing entirely with orthogonal projection onto learned subspaces (CapProNet). All four treat each image independently, so dropping the loop means dropping the refinement that clustering provides. We keep the refinement and pay for it with time, which is only possible because RL observations arrive as a correlated stream.

Two papers constrain the design. Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale finds that widening the encoder hurts under plain PPO unless exploration is improved. Any gain we report must therefore be parameter-matched and not a capacity effect. An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks finds that training encoders end-to-end under RL is unstable compared with a frozen pretrained ResNet18. This makes training stability a first-class outcome for us, not an afterthought.

The closest routing-for-control work is VER. VER routes among whole distilled foundation-model backbones for each sample. It does not route parts to objects inside a single encoder, and it does not study procedural-level generalization.

Research Program

The encoder is a small IMPALA-style convolutional stem. Its output feature map is reshaped into primary capsules, one per spatial cell. These are routed to a set of object capsules whose output vectors feed the policy and value heads. Each primary capsule $i$ produces a vote $\hat u_{j|i}^t$ for object capsule $j$ at timestep $t$. Coupling weights are a softmax over routing logits $b_{ij}^t$, and object capsules $v_j^t$ are the squashed weighted sum of votes. The key change is the logit update.

$$b_{ij}^t = \mathrm{sg}\!\left(b_{ij}^{t-1}\right) + \hat u_{j|i}^t \cdot v_j^{t,(0)}$$

Here $\mathrm{sg}$ is stop-gradient. $v_j^{t,(0)}$ is the object capsule computed from the carried couplings $\mathrm{softmax}(b^{t-1})$. We apply exactly one agreement update per frame. Logits reset to zero at episode start.

In a PPO rollout buffer, the carried logits are stored like a recurrent hidden state. This adds memory proportional to (primary capsules × object capsules) per environment. The stop-gradient keeps backpropagation within a single frame, so training cost per step equals that of a one-round router.

We will build three things. The first is this temporally amortized router. The second is its two anchors: a one-round router with no carry-over, following the inverted dot-product design, and a full three-round router run fresh on every frame. The third is a diagnostic harness. The harness measures how much routing mass lands on background pixels. It uses Procgen's rendering options, which can remove or restrict background themes, to obtain pixel-level foreground masks. It also measures convergence: how far the amortized couplings are from the three-round couplings on the same frame.

The central analysis tests whether routing mass on background predicts test-level return across seeds and games. If it does, the mechanism is doing what we claim. If returns improve but background mass does not fall, the gain comes from elsewhere, and we will say so.

Baselines

The naive reference is PPO with the standard IMPALA-CNN encoder, trained on the same fixed set of training levels and evaluated on unseen levels. It defines the generalization gap every other method must shrink.

The strongest reported existing method is the NeurIPS 2020 Procgen winner, Phasic Policy Gradient with random-translate augmentation. It reports a mean normalized reward of 0.6083 on the generalization track, against 0.2002 for the PPO baseline (Measuring Sample Efficiency and Generalization in RL Benchmarks). We will rerun it with our step budget and seeds rather than rely on the leaderboard number. Our capsule encoder will then be dropped into that exact pipeline, with the same algorithm, augmentation, steps, and a hyperparameter sweep of equal size. Beating it means a higher test-level mean normalized reward with non-overlapping 95% bootstrap intervals across at least five seeds. For the exploration-coupled scaling line, the reported figure is an optimality gap reduced from 0.58 to 0.36 by VSOP-3D+ (Improving Generalization on the ProcGen Benchmark). We treat that as a reference point, not a rerun target, because it changes the exploration algorithm, which is orthogonal to our question.

Four ablations isolate the key ingredient. A parameter- and FLOP-matched wider CNN tests whether capacity alone explains any gain. A plain attention-pooling head tests whether agreement matters at all. This head has learned object queries attending over the same primary features, with no vote-agreement term. A single-round capsule router without carry-over tests whether temporal amortization matters. A fresh three-round router tests whether amortization loses anything relative to full iteration; it is the upper bound on selectivity and the most expensive option. The claim succeeds only if the amortized router beats plain attention and one-round routing on background-shift test levels, matches three-round routing within noise, and runs at close to one-round throughput.

Why not just…?

Why not just use data augmentation? Random translation was decisive in the Procgen competition, and Cutout narrowed the CoinRun gap (Quantifying Generalization in Reinforcement Learning). The competition report also notes that augmentation is brittle and needs a fixed share of un-augmented frames. Augmentation does not help where failures stem from multi-object timing, as in Leaper and Dodgeball (Measuring Sample Efficiency and Generalization). We do not position capsules as a replacement for augmentation. Every comparison includes augmentation, so the question is whether routing adds to it. If augmentation alone matches the capsule agent on background-shift tests, the proposal's practical case fails, and the experiment will show it directly.

Why not just scale the CNN? Width helps in projected Gym tasks (Observational Overfitting in Reinforcement Learning), but it hurts under PPO without better exploration (Improving Generalization on the ProcGen Benchmark). The parameter- and FLOP-matched wider CNN ablation settles whether any capsule gain is really a capacity gain.

Why not just use plain attention or a transformer head over CNN features? This is the most serious threat. Learned queries can also concentrate on foreground cells, and non-iterative capsule routers are already close to attention. The only thing separating our router is the agreement term, refined over time. If plain attention pooling matches us on background-mass and test return, the contribution collapses to "attention helps," and we will report that. This is why the ablation exists.

Why not just freeze a pretrained encoder? A frozen ResNet18 stabilized pixel-based multi-agent training (An Extended Benchmarking of MARL Algorithms), so it is a fair competitor. It still left spatial-coordination tasks unsolved, and ImageNet features are not tuned to reject game backgrounds. We include it as a baseline on the natural-video setting, where pretrained features might plausibly be most robust.

Proposed Experiments

Diffuse-shift experiment. The first experiment uses the Procgen games with background themes. Agents train on a fixed set of levels with restricted themes and are tested on unseen levels with full backgrounds. We report the test mean normalized reward, the train–test gap, background routing mass, and environment steps per second.

Continuous-control check. The second applies the same comparison to DeepMind Control tasks with natural video backgrounds (Natural Environment Benchmarks for Reinforcement Learning). No reported result was found on Lacuna for this benchmark, so all numbers will be our own reruns.

Compact-distractor control. The third reproduces the Sonic setting (Observational Overfitting in Reinforcement Learning). There IMPALA test reward rose from 1130 ± 40 to 1250 ± 40 when the timer and score were blacked out. We predict that the capsule agent will not recover this gap on its own, and that routing mass on the timer will stay high. If it does close the gap, our proposed mechanism is wrong about what routing rejects, even if the result looks good.

Decisive checks. Three results would settle the proposal. The amortized router must beat the no-agreement attention head on diffuse-shift test return. It must track three-round couplings within a small divergence after a few frames of warm-up. Its background routing mass must correlate negatively with test return across seeds. Any one of these failing would mean the claim is wrong in a specific way.