Letting Shared Perception Keep Learning Without Breaking Its Consumers

A modular agent is built from separate parts: a perception module that turns camera images and instructions into features, and several downstream parts that read those features, such as skill-specific action heads and the small classifiers that decide which skill to run. Such an agent faces a problem that neither of the two fields studying it handles on its own. People who build integrated agents know that when one module is updated, the modules downstream of it silently break. People who study continual learning know that a network retrained on new tasks overwrites old ones, which is called catastrophic forgetting. Current robot continual learners avoid both problems by freezing the shared perception module and only adding small new parts. The cost is that the agent can never improve the representation all of its skills share.

We propose to let the shared perception module keep learning. Its changes would be limited only in the directions that some existing downstream consumer actually reads. Every old action head and every old selection classifier contributes to a sensitivity-weighted penalty on how much the features may drift. Directions no consumer is sensitive to stay free. The claim to test is that this "consumer-anchored interface" keeps forgetting as low as a frozen backbone while recovering the forward transfer that anti-forgetting methods are known to suppress. Forward transfer here means learning new skills faster because of old ones.

Core Claim and What Is New

Two facts from the file frame the opportunity. CLARE adds lightweight adapters to a frozen vision-language-action model and picks among them using small autoencoders. It forgets almost nothing and reaches an area-under-curve of 75.1±1.3 on LIBERO-Long, against 52.9±1.6 for experience replay. Its ablations also show that the modules that fuse vision and language matter more for retention than the action heads do. LIBERO documents the "forward transfer paradox": EWC and replay learn new manipulation skills worse than naive sequential fine-tuning. It also shows that supervised pretraining on LIBERO-90 can make the representation rigid.

Taken together, these suggest that the frozen upstream representation is where both retention and plasticity are decided. Freezing it trades one for the other. The architecture literature explains why unconstrained updates fail. Uhr and Kochen's MIKROKOSMS and Robots found that linking separate programs broke down without shared representations. The "interface brittleness" problem in the file says pipelines break when an upstream module changes. We read continual forgetting in a modular agent as exactly this interface failure. The upstream change is harmless unless it moves features along directions a consumer uses.

Let $f_\phi$ be the shared perception module with parameters $\phi$, where $\phi_{\text{old}}$ are its parameters before the current task. Let $g_k$, for $k=1..K$, be the existing consumers: frozen skill adapters and heads, plus their routing autoencoders. On inputs $x$ drawn from the current task only, we train with

$$ \mathcal{L}(\phi) = \mathcal{L}_{\text{task}}(\phi) + \lambda \, \mathbb{E}_{x}\sum_{k=1}^{K} \big\| J_k(x)\,\big(f_\phi(x) - f_{\phi_{\text{old}}}(x)\big) \big\|^2 . $$

Here $J_k(x)$ is the Jacobian of consumer $k$'s output with respect to its input features, evaluated at the old features $f_{\phi_{\text{old}}}(x)$. For routers, the output is the reconstruction error, so the router's view of "which skill is this" is protected along with the skills themselves. The penalty is the first-order change in every consumer's output. Features can move freely in the null space shared by all consumers.

This differs from the closest work in specific ways. CLARE never updates the base, and its routers are trained on fixed features; we update the base and treat the routers as consumers whose inputs must be protected. Hierarchical Orchestra of Policies freezes whole policies and blends their actions, so it shares no representation across skills at all. Approximating Function Space Distance for Continual Learning in Transformers bounds drift of a single network's outputs uniformly. We bound drift only where a set of heterogeneous consumers is sensitive, and the consumer set grows over time. Additive Parameter Decomposition repairs past-task residuals after the shared weights move. It needs task identities at test time and has only been run on classification. We prevent the harmful movement instead of repairing it afterward, and we need no task identity.

Research Program

We build a two-level agent. The top level is a shared perception and fusion module initialized from a pretrained model, and it stays trainable, possibly through a low-rank update so that compute stays comparable to adapter methods. The bottom level is a growing library of skill adapters with autoencoder routers, borrowing CLARE's novelty-triggered expansion and reconstruction-error routing so that the only change from CLARE is the plastic, anchored upstream.

On each new task, the new adapter and the upstream module are trained jointly under the objective above. Anchoring uses only current-task observations as probe inputs. This keeps the method free of stored examples and sidesteps the result in Replay can provably increase forgetting that small replay buffers can turn vanishing forgetting into constant forgetting. LIBERO scenes share objects and layouts across tasks, so current-task images plausibly cover the feature regions old consumers use. Whether this holds is itself something we measure, by comparing against a variant that anchors on a small stored buffer.

Three analyses accompany the method. The first logs interface drift per consumer, the Jacobian-weighted term above, alongside each old skill's success rate, to test whether drift predicts forgetting better than raw feature distance does. The second tracks router accuracy separately from skill accuracy. This matters because we expect a plastic upstream without router anchoring to fail mainly by misrouting, which is a failure mode unique to integrated systems. The third breaks down forward-transfer gains by LIBERO suite. The declarative/procedural split motivates the prediction that a plastic upstream helps most on LIBERO-Object, where new object concepts must enter the shared representation. That split is drawn from Langley's Progress and Challenges in Research on Cognitive Architectures and from LIBERO's own suite design. On LIBERO-Goal, which varies behavior over fixed objects, we expect little difference from a frozen upstream.

Baselines

The naive reference points are sequential fine-tuning of the whole policy and sequential fine-tuning with low-rank adapters. LIBERO shows plain sequential fine-tuning has the best forward transfer among its classic methods but heavy forgetting. It therefore sets the plasticity ceiling we hope to approach.

The strongest existing method is CLARE, run with its published expansion threshold and routing. It is the strongest competitor because it already achieves near-zero negative backward transfer without stored examples on the same benchmark. Its reported number to beat is LIBERO-Long AUC 75.1±1.3 CLARE. We will rerun it rather than copy it, with the same pretrained base, demonstrations, number of gradient steps, and hyperparameter search budget as our method. We also match the total trainable parameter count, since our upstream update adds parameters that CLARE lacks. Beating it means higher AUC and higher forward transfer on LIBERO-Long and LIBERO-Object, with negative backward transfer that is not statistically worse, across at least three seeds and two task orders.

Classic anti-forgetting methods come next. These are experience replay, EWC, and PackNet, using LIBERO's reported PackNet forgetting as a reference point: negative backward transfer of 0.06 on LIBERO-Goal and 0.07 on LIBERO-Spatial LIBERO.

The ablations that isolate the key ingredient each keep the plastic upstream and the adapter library. The first replaces the consumer-weighted penalty with uniform feature distillation, which is FSD-style anchoring applied to features. The second drops the routers from the consumer set. The third replaces the Jacobian penalty with full output distillation through every old consumer. The fourth removes anchoring entirely. If uniform anchoring matches our method, the sensitivity weighting is not the contribution. If output distillation matches it, the linearization is only a cost saving, and the finding becomes where to anchor rather than how.

Why not just…?

Why not just keep the backbone frozen and add adapters, as CLARE does? This is the real threat. If the pretrained representation already contains what later LIBERO tasks need, a frozen base loses nothing and is simpler. The file gives only indirect evidence against it. LIBERO reports that pretraining on LIBERO-90 can harm later lifelong learning, and CLARE itself notes that adapters cannot compensate for missing primitives in the base. The direct test is the suite breakdown. If our method gains no forward transfer over CLARE on LIBERO-Object and LIBERO-Long at matched parameters, we will conclude that freezing is sufficient and report that.

Why not just replay a buffer of old demonstrations? Replay reaches 52.9±1.6 AUC on LIBERO-Long against CLARE's 75.1 CLARE. Mahdaviyeh et al. prove small buffers can increase forgetting Replay can provably increase forgetting. We still include a buffer-anchored variant, because replay used only as probe inputs for the anchor, rather than as training targets, is a different and possibly benign use.

Why not just regularize the whole network with EWC or output distillation? EWC suppresses forward transfer in LIBERO. On split CIFAR-100 it trails parameter decomposition, 53.7% against about 60.7% Additive Parameter Decomposition. Output distillation through old consumers is closer to our method and is included as an ablation. If it wins, the proposal reduces to "distill through consumers, including routers," which would still be a useful result.

Proposed Experiments

All runs use the LIBERO suites: Spatial, Object, and Goal with 10 tasks each, and LIBERO-Long for long sequences. We use LIBERO's metrics of forward transfer, negative backward transfer, and AUC, plus router accuracy, trainable parameters, and inference latency. CLARE reports an overhead under 3 ms and about 2% extra GPU memory per task, which our upstream update must not exceed by much. For task-order robustness we report the spread across orders, the same quantity APD measures as order disparity.

Three results are decisive. The first is whether our method raises LIBERO-Long AUC and LIBERO-Object forward transfer over the matched CLARE rerun without increasing negative backward transfer. The second is whether removing routers from the consumer set produces forgetting that is mostly misrouting, which would confirm that interface drift is the mechanism. The third is whether Jacobian-weighted drift predicts per-skill forgetting better than unweighted feature drift. A negative answer to the first, with positive answers to the second and third, would still establish the diagnostic. It would, however, sink the claim that a plastic shared perception module is worth having.