Residual Ledgers: Replay-Free Backward Transfer in Factored Robot Policies

Robots that learn tasks one after another usually aim only to not forget. Old skills are frozen, protected by penalties, or rehearsed from stored data. Each of these approaches treats a new task as a threat to old ones. This note proposes a way for new tasks to actively improve old ones, without storing past experience, by building each task's policy from a shared set of weight vectors. In a factored policy, each task's network weights are a task-specific mix of a few shared basis vectors: $\theta_t = \Phi w_t$, where $\Phi$ is a $d \times k$ matrix of $k$ shared "weight primitives" and $w_t$ is a $k$-dimensional recipe for task $t$. A task trained only through its recipe stops improving once its recipe is optimal, but its full weight vector is usually not optimal. The gradient of its return with respect to the full weights is still nonzero in directions the basis cannot yet reach. We propose to store that leftover gradient, one vector per task, in a "residual ledger". When later tasks reshape the basis, the ledger tells us, to first order, which basis updates also help each earlier task.

The core claim is concrete. In a factored actor, the end-of-task residual gradient carries a usable signal for backward transfer. In an ordinary network trained to convergence, the same gradient is close to zero. Steering basis updates toward agreement with this ledger should give positive, statistically reliable backward transfer on Continual World. The method needs no replay buffers, no Hessians, and no old-environment interaction, and it costs $O(d+k)$ memory per task.

What exists and what is missing

Factored policies already have a lifelong-learning history. PG-ELLA learns a shared basis $L$ online and notes that updating $L$ changes old tasks' effective policies. However, it summarizes each old task by a second-order Taylor expansion around a separately trained optimum. That requires per-task Hessians and was only shown on shallow policies for systems such as cart-pole and quadrotor control. LPG-FTW removes the train-to-convergence step by optimizing task coefficients directly with on-policy policy gradients. It shows no forgetting on MuJoCo and Meta-World, but it does not demonstrate improvement of old tasks above their original level. Its quadratic surrogate penalizes deviation and never asks whether a deviation would help. PaCo supplies the deep, off-policy version of the factorization, $\theta = \Phi w$ inside multi-task Soft Actor-Critic (SAC). It reaches 85.4% success on MT10-rand with $K=5$, which shows that manipulation policies lie on a small parameter manifold. PaCo trains all tasks at once, though, and never studies sequential task streams. We reuse PaCo's parameterization and its recipe-reset trick as the method substrate. LPG-FTW and PG-ELLA serve as the closest competitors.

On the transfer side, CUBER gets small positive backward transfer in supervised vision, +0.13% on Split CIFAR-100 and +0.23% on Split MiniImageNet. It updates shared layers along new-task gradients when they correlate positively with old-task subspaces, which requires layer-wise SVDs at task boundaries. Three other works show backward transfer through a shared factored store, but each in a setting without online exploration. DF-CNN and the what/how RNN of Shan et al. do it in supervised vision and synthetic cognitive tasks. HTAC reports negative forgetting in offline continual RL, routed by language embeddings. Together these motivate the proposal: shared factored knowledge can produce reverse transfer. None operates in online actor-critic control, where the agent collects its own shifting data.

Two results constrain the design. Continual World finds that standard continual-learning methods almost never show positive backward transfer in RL, and that resetting buffers hurts. Disentangling Transfer in Continual RL finds that the critic drives most forward transfer, and that regularizing the critic or freezing shared backbones damages learning. We therefore leave the critic fully plastic and put all backward-transfer machinery on the factored actor. The ledger acts as a frozen first-order imprint of what each old critic "thought" at the end of its task. That imprint lets the live critic stay unconstrained.

Core mechanism

Let $J_\tau(\theta)$ be task $\tau$'s SAC actor objective. When training on task $\tau$ ends, its recipe $w_\tau$ is approximately stationary, so $\Phi^\top \nabla_\theta J_\tau(\Phi w_\tau) \approx 0$. The residual gradient $r_\tau = \nabla_\theta J_\tau(\Phi w_\tau)$, however, is generally nonzero. We estimate $r_\tau$ by averaging actor gradients over many minibatches from task $\tau$'s buffer before discarding it. We then store $r_\tau$ and $w_\tau$.

Any later basis change $\Delta\Phi$ moves task $\tau$'s weights by $\Delta\Phi\, w_\tau$. Its first-order effect on the old task is therefore

$$\Delta J_\tau \approx \langle r_\tau, \Delta\Phi\, w_\tau \rangle = \langle G_\tau, \Delta\Phi \rangle, \qquad G_\tau = r_\tau w_\tau^\top,$$

where $G_\tau$ is a rank-one $d\times k$ matrix and $\langle\cdot,\cdot\rangle$ is the elementwise inner product. During task $t$, the basis update solves a small problem over the ledger:

$$\Delta\Phi^\star = \arg\max_{\Delta\Phi}\ \langle G_t, \Delta\Phi\rangle + \lambda \sum_{\tau<t}\langle G_\tau, \Delta\Phi\rangle \quad \text{s.t.}\ \langle G_\tau, \Delta\Phi\rangle \ge 0\ \forall \tau,\ \ \|\Delta\Phi\, w_\tau\|\le \epsilon_\tau,$$

Here $G_t$ is the current new-task gradient with respect to $\Phi$ and $\lambda$ weights the backward-transfer bonus. The bound $\epsilon_\tau$ limits how far task $\tau$'s weights may move from the point where its residual was recorded, which keeps the linearization valid. That drift is computable exactly from $\Phi$ and $w_\tau$, with no data.

The constraints are the analogue of CUBER's no-forgetting regime, now expressed in basis space and costing $O(T)$ inner products. The $\lambda$ term is the backward-transfer regime: it pushes the basis along directions that old tasks wanted but could not reach. When a task's drift budget is spent, its constraint is enforced as a hard projection. Its recipe can be refreshed only if the protocol permits revisiting, as in CW20.

The key point, testable on its own, is that this signal exists only because of the factorization. Residual gradients in an unfactored SAC actor at convergence should be near noise level.

Research Program

We build sequential PaCo: a SAC agent on Continual World with a factored actor ($k$ swept over 5 to 20), task-specific recipes, PaCo's recipe reset for unstable tasks, and a shared, fully plastic critic with task-specific output heads, warm-started across tasks. We then add the residual ledger and the constrained basis update.

The analysis has three parts. First, we measure the norm and signal-to-noise ratio of $r_\tau$ in factored versus unfactored actors as a function of $k$. Second, we check whether the predicted change $\langle G_\tau,\Delta\Phi\rangle$ correlates with the measured change in old-task return, obtained from evaluation-only rollouts. Third, we test how prediction accuracy decays with drift $\|\Delta\Phi\, w_\tau\|$, which sets $\epsilon_\tau$.

As a secondary question tied to fixed basis size, we treat persistent infeasibility as an expansion signal. If a new task's gradient cannot be followed without violating ledger constraints, a new basis column is added. Infeasibility here means $\langle G_t, \Delta\Phi\rangle$ stays near zero under the constraints.

Baselines

The naive reference is fine-tuning a single SAC network across the stream. Continual World reports a forward transfer of 0.20 for it, with near-total forgetting. We also run PackNet, which Continual World reports at 0.80 average success. It isolates weights per task and therefore has backward transfer bounded at zero, so it marks the "stable but one-way" point.

The strongest existing methods are ClonEx-SAC, which reports 87% final success and forward transfer of 0.54 on Continual World, and SSDE, which reports 95% success on CW10-v1 (versus 86% for ClonEx-SAC in that paper). ClonEx-SAC is the strongest competitor on the backward-transfer question because it uses replay plus behavioral cloning, a strictly larger information budget than ours. SSDE is strongest on raw success. We found no reported backward-transfer number on Lacuna for either method on Continual World, so we measure it ourselves.

The closest factored competitor is an LPG-FTW-style update ported to SAC. It applies a quadratic deviation penalty on $\Phi w_\tau$ with a diagonal curvature estimate, which is the cheapest faithful version of LPG-FTW's second-order surrogate. A second factored baseline is sequential PaCo with an unconstrained basis.

The ablations isolating the key ingredient are four:

  • constraints only, with $\lambda=0, which should give stability but no positive backward transfer;
  • a ledger with $r_\tau$ replaced by random vectors of matched norm;
  • a ledger built on an unfactored actor, applying CUBER-style alignment to full weights;
  • a ledger that is never refreshed, compared with one bounded by $\epsilon_\tau$.

All methods share network width, SAC hyperparameters, environment steps per task (1M, the Continual World protocol), and the CW10, CW20, and triplet sequences. Each method gets the same hyperparameter search budget on one held-out triplet. ClonEx-SAC keeps its replay, and we report its memory cost alongside.

Success means three things. Backward transfer must be positive with a 95% confidence interval excluding zero across at least 10 seeds on CW10. Average success must be within a few points of ClonEx-SAC's 87%. Forward transfer must not fall below sequential PaCo's. We do not expect to beat SSDE's 95% on raw success. The claim is a qualitatively different outcome, old tasks getting better, at replay-free memory cost.

Why not just…?

Why not just use replay with behavioral cloning, as ClonEx-SAC does? ClonEx-SAC's 87% success and 0.54 forward transfer make it a real threat on headline success. Its cloning loss, however, pulls the policy back toward old actions, which defends old performance rather than improving it. It also stores transitions from every task. We will measure its backward transfer directly. If it already shows positive backward transfer comparable to ours, our contribution narrows to "the same effect without replay," and we will say so.

Why not just port LPG-FTW's second-order surrogate to SAC? A quadratic penalty around $\Phi w_\tau$ only discourages movement. Its linear term is assumed zero, so by construction it cannot predict improvement. The constraints-only ablation versus the full ledger, and the LPG-FTW-style baseline versus the ledger, test exactly whether the first-order residual term adds backward transfer.

Why not just apply CUBER-style gradient alignment to an ordinary actor? In an unfactored network trained to convergence, the old-task gradient is near zero. There is little to align with, and CUBER's gains in vision were tenths of a percent. The unfactored-ledger ablation, together with the residual signal-to-noise measurement, tests whether factorization is what makes the signal usable. If unfactored residuals turn out equally informative, the factorization is not the key ingredient and the proposal should be reframed.

Why not just freeze the basis after the first few tasks? Freezing gives zero backward transfer and limits later tasks to an early span. Disentangling Transfer reports that frozen shared backbones hurt RL. We include frozen-$\Phi$ as an extra point on CW20.

Decisive checks

The idea fails quickly if any of three things happens. First, $r_\tau$ in the factored actor may be no larger than in the unfactored one. Second, predicted and measured old-task changes may be uncorrelated even at small drift. Third, positive backward transfer may vanish once the random-ledger control is matched.

The idea succeeds if all three of the following hold. Residuals must be clearly larger in factored actors at small $k$. The first-order prediction must track measured change within the $\epsilon_\tau$ budget. On CW10 and CW20, the full method must show significantly positive backward transfer while the constraints-only and random-ledger variants show none. The CW20 repeat gives a further test: on the second pass, tasks should start from a higher success rate than they ended the first pass with. That is the clearest behavioral signature that later learning improved earlier skills.