Backward Value Propagation That Knows Which Physics It Came From

Robots trained in simulation usually reach the real world through domain randomization: every training episode draws new physical parameters (mass, friction, motor gain, latency) so that the real robot looks like one more random draw. Separately, a family of reinforcement learning methods speeds up learning by updating value estimates backward along stored episodes. Instead of letting a reward trickle back one step per update, these methods sweep from the end of an episode to its start, so a reward found at step 200 reaches step 1 in one pass. The two ideas have never been combined, and combining them naively should fail. A trajectory recorded under low friction may be impossible under high friction. Chaining its rewards backward into a value function that does not know which friction produced it teaches the value function to expect returns that most randomized worlds cannot deliver.

We propose to give the value-learning network (the critic) the episode's randomized physics parameters, which the simulator knows but the deployed policy never sees. Inside one fixed physics instance, a recorded trajectory is a plan that is actually achievable. Its backward-chained return is therefore a valid lower bound on the best achievable value for that instance, and taking the maximum of this bound and the usual one-step estimate is safe. Across instances, the same maximum is not safe. The core claim is an interaction effect. Under domain randomization, backward propagation should help only when the critic is conditioned on the physics parameters, and should hurt more as the randomization range widens when it is not. The actor still sees only onboard observations, so the policy remains deployable zero-shot.

Core Claim and Mechanism

Three bodies of prior work set this up. Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update (EBU) samples whole episodes and computes targets from the final step back to the first. It uses a blending factor $\beta$ to limit overestimation and matched DQN's mean Atari score with 5% of the samples. Its evaluation is a single, unchanging environment per game. Look Back When Surprised (Introspective Experience Replay, IER) stabilizes reverse replay for neural networks. It picks high-error transitions as "pivots" and replays the steps leading up to them. It beat uniform, prioritized and hindsight replay on 11 of 13 tasks, again without randomization. Understanding Domain Randomization for Sim-to-real Transfer shows that domain randomization turns training into a latent MDP, meaning a decision problem whose physics parameter $\xi$ is fixed within an episode but hidden. It also shows that the deployed policy needs memory to infer $\xi$. That paper motivates our concern: transitions depend on $\xi$, so backward chains are only coherent within one $\xi$. EBU and IER supply the mechanism we reuse.

For continuous control we adapt EBU's recursion to an off-policy actor-critic. For an episode with physics $\xi$, states $s_t$, actions $a_t$, rewards $r_t$ and discount $\gamma$, we compute targets from $t=T$ down to $1$:

$$y_t = r_t + \gamma \max\Big(\beta\, y_{t+1} + (1-\beta)\, \bar Q(s_{t+1}, a_{t+1}, \xi),\ \ \bar Q(s_{t+1}, \pi(h_{t+1}), \xi)\Big)$$

Here $\bar Q$ is the target critic, $\pi(h_{t+1})$ is the actor's action given its observation history $h_{t+1}$, and $\beta\in(0,1)$ is EBU's blending factor. The first argument of the max is the value of continuing along the recorded path. The second is the value of switching to the current policy. Suppose the simulator is deterministic given $\xi$ and the recorded actions. Then the recorded continuation is realizable, so the first term with $\beta=1$ lower-bounds the optimal value $Q^*(s_t,a_t,\xi)$. The max can then only correct underestimates and never invents unreachable returns. If $\xi$ is removed from the critic's input, that guarantee disappears: the chained return from one instance is credited to states shared by all instances. This is exactly the target inflation the latent-MDP view predicts.

The privileged-critic literature constrains the design. Asymmetric Actor Critic for Image-Based Robot Learning gave the critic full simulator state while the actor saw images, and it transferred 5/5 to a real Fetch arm, using one-step targets. Unbiased Asymmetric Actor-Critic shows that a critic conditioned only on state is biased when the actor depends on history. For this reason the policy-action term above uses $\pi(h_{t+1})$, and we test a variant whose critic also receives the actor's recurrent state. Informed Asymmetric Actor-Critic finds that irrelevant privileged inputs add structured noise. So we condition on the randomized parameters $\xi$ rather than dumping in every simulator variable, and we ablate which components of $\xi$ matter.

Research Program

The first stage is diagnostic and needs no new algorithm. We take EBU-style and IER-style replay as published and add them to soft actor-critic under dynamics randomization of increasing width. We measure two quantities. The first is target inflation: the gap between backward-chained targets and Monte Carlo returns re-rolled under freshly sampled $\xi$. The second is IER's pivot purity: the fraction of top-error pivots whose error is explained by $\xi$ rather than by unlearned value. We obtain the second by checking whether the pivot's error collapses when a $\xi$-conditioned critic scores it. This tests a hypothesis the research file raises but no paper checks: under randomization, "surprise" is partly physics variation, which would send reverse replay chasing noise.

The second stage builds the $\xi$-conditioned backward update above. It has a recurrent actor, as Chen et al. require for transfer, and a critic that receives $(s,\xi,a)$. Optionally the critic also receives the actor's history, and optionally pivot selection follows IER. We study $\beta$ schedules, because EBU reports that high $\beta$ helps early and hurts later. We also study the effect of simulator stochasticity on the lower-bound argument by injecting contact and sensor noise and measuring how often the max selects an unrealizable branch.

The third stage is transfer. Following Kadokawa et al., we use a held-out "real" simulator outside the training range as a cheap surrogate. If hardware access is available, we then run a real arm task with sparse success reward. Hardware is a stretch goal, and the claims stand on the simulated evidence alone.

Baselines

The naive reference point is soft actor-critic with domain randomization and uniform replay, the SAC-DR setup from Cyclic Policy Distillation. That paper reports it failing to learn HalfCheetah under wide randomization. Reproducing that failure is our first sanity check.

The strongest existing method for sample-efficient learning under wide randomization is Cyclic Policy Distillation (CPD). CPD splits the parameter range into overlapping sub-domains, trains local policies cyclically and distills them into one global policy. It reports 3x to 6x better sample efficiency than P2PDRL and Divide-and-Conquer on Pendulum, convergence on HalfCheetah where SAC-DR failed, and 40% higher reward than baselines on a physical UR3 ball-dispersal task. We run CPD with its published partitioning on the same randomization ranges, the same environment-step budget and the same hyperparameter-search budget. Beating it means reaching CPD's final return in fewer environment steps on Pendulum and HalfCheetah, and matching or exceeding it on the held-out out-of-range physics. CPD does not rule out our method; the two can stack. We also run CPD with our backward targets inside each sub-domain, which addresses the file's Gap 2 directly.

The strongest existing backward-propagation method is IER, applied unchanged under randomization. No reported result was found on Lacuna for IER under domain randomization; its 11/13 result is for fixed environments. Beating it means higher return at equal steps once the randomization width exceeds a threshold, which we predict exists.

The ablation that isolates the key ingredient is our backward update with $\xi$ removed from the critic. A second ablation is the $\xi$-conditioned critic with ordinary one-step targets, which is Pinto et al.'s asymmetric critic with $\xi$ as the privileged input. The claim survives only if the full method beats both, and only if the unconditioned backward variant degrades with randomization width while the conditioned one does not. All methods share network sizes, replay capacity, environment steps, five or more seeds, and an equal random-search budget over learning rate and $\beta$ or $n$.

Why not just…?

Why not just condition the critic on $\xi$ and use $n$-step returns or prioritized replay? This is the most serious threat. A $\xi$-conditioned critic with $n$-step returns removes most of the cross-instance inconsistency, and it is simpler. EBU reports beating both prioritized replay and $n$-step DQN on Atari, but that evidence comes from a fixed environment. So we include $\xi$-conditioned $n$-step returns, tuned over $n$, and $\xi$-conditioned Peng-style $\lambda$-returns as baselines. Our recursion without the max reduces to such a $\lambda$-return. The proposal's distinctive content is therefore the max over a certified lower bound, and it must beat these on long-horizon sparse tasks to matter. If it does not, the honest conclusion is that conditioning is what matters and backward sweeps add nothing, and the diagnostic stage still stands as a result.

Why not just use Cyclic Policy Distillation? CPD handles the instability of wide ranges by narrowing each learner's range. It does not change how a reward travels back through a long episode within a sub-domain. On sparse-reward manipulation, slow credit assignment remains the bottleneck. SURREAL reports Bin Picking and Nut-and-peg Assembly unsolved even with 64 actors (SURREAL). The head-to-head and the stacked CPD-plus-backward run will show whether the two benefits are separate or redundant.

Why not just use hindsight relabeling? Hindsight relabeling creates reward signals by pretending the reached state was the goal, but it still propagates those rewards with one-step targets. It is orthogonal to the backward sweep, so we combine the two on goal-conditioned tasks rather than treat relabeling as a rival. We also report it as a baseline on its own, since IER found it weaker on FetchReach.

Proposed Experiments

The environments are randomized Pendulum and HalfCheetah, using CPD's protocol and ranges plus a doubled range. We add randomized FetchReach, which IER used, and robosuite Block Lifting and Pick-and-Place under operational space control with sparse success reward (robosuite). Mass, friction, damping, actuator gain and latency are randomized. The metrics are return or success versus environment steps, wall-clock time, and target inflation measured against re-rolled Monte Carlo returns. Transfer is reported as performance on out-of-range held-out physics, including a "real" variant built the way RGAT builds surrogate targets (Reinforced Grounded Action Transformation).

The decisive check is a two-by-two over critic conditioning ($\xi$ or not) and target type (backward or one-step), swept across randomization width. The hypothesis is confirmed if backward targets help with $\xi$-conditioning at every width but hurt without it as width grows. Measured target inflation should track the performance gap. A second decisive check is the comparison against tuned $\xi$-conditioned $n$-step and $\lambda$-returns on Pick-and-Place. If the max-with-lower-bound gives no gain there, we drop the algorithmic claim. A third check concerns violations of the "real world is contained in the simulator's range" assumption that Chen et al. require. As held-out physics moves outside the training support, we test whether the $\xi$-conditioned critic's sharper values make the actor more brittle than SAC-DR's blurrier ones. A positive answer would be a meaningful negative result for privileged backward targets.