Reinforcement learning agents usually need a hand-written scalar reward. For long, multi-stage tasks those rewards are brittle and easy to exploit. Separately, a line of neuroscience-inspired work has shown that networks which simply "settle" into a low-energy state can compute the same weight updates as backpropagation using only signals available at each synapse. This proposal joins the two. We treat a short future trajectory, made of states and actions over a planning window, as the state of one energy-based network. The task is given only as a goal observation clamped at the end of that window, never as a reward. Planning is the network relaxing until a feasible path to the goal appears. Learning is the same network being gently pushed toward trajectories that were actually observed. Every weight change comes from local quantities.
The concrete bet is that this matches, and on stitching-heavy tasks may beat, goal-conditioned supervised policies on offline benchmarks. The reason is structural. The trajectory energy is a sum of local transition terms, so a low-energy path can pass through segments that never co-occurred in the data. It does this without a reward, without a value function, and without backpropagation. If the bet fails, the failure is informative in a specific way: it would show that the exactness results for these learning rules do not survive closed-loop, contact-rich control.
Core Claim
Equilibrium propagation (EP) trains a recurrent energy network in two phases. In the free phase the network settles given its input. In the nudged phase a small force of strength $\beta$ pulls its outputs toward a target, and the difference between the two settled states yields the gradient Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation. That difference matches recurrent backpropagation Equivalence of Equilibrium Propagation and Recurrent Backpropagation and backpropagation through time for static inputs Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input.
The research file identifies two obstacles to using EP for control. First, nobody has said what should play the role of the nudge when no supervisor exists. Second, settling seems incompatible with a world that keeps moving.
We answer both with one move. The input to the network is a whole planning window $\tau = (x_0, a_0, x_1, \dots, a_{H-1}, x_H)$, where $x_t$ are states, $a_t$ are actions, and $H$ is the horizon. The current state $x_0$ and the goal $g$ are clamped, and the window is re-solved at every control step. This makes the "static input" a receding-horizon plan, much as Sequence Learning Using Equilibrium Propagation froze text into a static representation. The energy is
$$E_\theta(\tau; g) = \sum_{t=0}^{H-1} \phi_\theta(x_t, a_t, x_{t+1}) + \tfrac{\lambda}{2}\lVert x_H - g\rVert^2 ,$$
where $\phi_\theta$ is a learned local transition energy (low for physically plausible steps) and $\lambda$ weighs the goal term. Following Active Inference or Control as Inference? A Unifying View, that goal term is a Gaussian "observation" of the desired future. In that paper the same correspondence recovers exact quadratic-regulator control in linear systems, so the goal here is evidence to be explained, not a reward to be maximized.
Training uses offline trajectory segments with hindsight goals: $g$ is set to the segment's final state, so no reward is ever defined. In the nudged phase the free nodes are pulled toward the recorded states and actions by a cost $\beta C$, and the weights change by
$$\Delta\theta \propto -\frac{1}{\beta}\left(\frac{\partial E_\theta}{\partial \theta}(\tau^{\beta}_{*}) - \frac{\partial E_\theta}{\partial \theta}(\tau_{*})\right),$$
where $\tau_*$ and $\tau_*^\beta$ are the free and nudged equilibria. Each term involves only one transition's variables, so the update is local in both space and time.
At test time no weights change. The agent clamps $x_0$ and the commanded goal, settles, executes $a_0$, and replans. Because $\phi_\theta$ penalizes infeasible transitions, the agent cannot satisfy the goal term by "teleporting." This gives a built-in check against the cheapest kind of goal-distance hacking, though not a guarantee.
Relation to Prior Work
Several works motivate this proposal by replacing rewards with predictive or goal structure while keeping global backpropagation:
- Goal-Conditioned Predictive Coding for Offline Reinforcement Learning compresses masked future trajectories with a Transformer trained by backpropagation and feeds a summary to a policy network. We keep its offline, goal-conditioned setting but replace the Transformer with an energy network that is both the planner and the learner.
- Predictive Coding for Locally-Linear Control uses contrastive coding as a representation loss and then plans with a regulator that needs a quadratic cost and smooth dynamics. Our planner is the relaxation itself and makes no linearity assumption.
- R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models removes reward shaping through learned preferences, but trains its world model and actor-critic with backpropagation through time and needs demonstration seeds.
Other work supplies the method or constrains the design. Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections provides residual and feedback tricks we reuse for stable settling. Equilibrium Propagation for Non-Conservative Systems warns that dynamics which are not energy-conserving break the basic derivation. That is why we write the controller's internal dynamics as a true energy descent even though the physics it models is not conservative. Habitual and Reflective Control in Hierarchical Predictive Coding shows that a predictive coding hierarchy can develop fast "habitual" shortcuts alongside slow deliberation, but only on static MNIST choices. We test the deliberative half on real sequential control, and optionally distill a fast first-action readout.
The local-learning literature has so far stayed on static classification. This includes the exact predictive-coding equivalence result [Can the Brain Do Backpropagation?
- Exact Implementation of Backpropagation in Predictive Coding Networks](/work/can-the-brain-do-backpropagation-exact-implementation) and deep scaling via $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks. We do not claim to scale deeper. We claim a new objective class: goal-clamped trajectory energies.
Research Program
The first step builds the transition energy $\phi_\theta$ as a small symmetric network over consecutive state–action–state triples. It optionally adds a Hopfield-style memory term, borrowed from the EP sequence work, that pulls window states toward stored dataset states, which encourages paths to stay on the data manifold.
The second step characterizes the two quantities that local learning trades away. The first is the nudge strength $\beta$, measured by how closely EP updates align with gradients obtained by implicit differentiation of the same energy. The second is the number of settling iterations. Each is measured as a function of contact events in the window, because the equivalence proofs assume smooth converged equilibria and contacts are where we expect them to break.
The third step is the main offline evaluation. The fourth is a closed-loop online stage on Meta-World. There, transitions gathered by the planner are added to the training buffer with hindsight goals, and the only task signal is a goal state together with the benchmark's binary success flag, which is used purely for evaluation.
A known risk is mode averaging. When the data contain several routes between the same endpoints, the squared nudge cost pulls toward their mean. We will measure this directly on AntMaze, where alternative routes exist, and if needed replace the nudge cost with a nearest-segment cost.
Baselines
The naive reference point is goal-conditioned behavioral cloning, RvS-G: a multilayer perceptron regressing actions from (state, goal) on hindsight-relabeled data. Goal-Conditioned Predictive Coding for Offline Reinforcement Learning reports it at 26.4 normalized score on AntMaze-Ultra and 71.7 on FrankaKitchen-Partial. We rerun it on our exact data splits and hindsight-relabeling scheme, with the same hyperparameter search budget as our method.
The strongest existing reward-free offline method in the file is GCPC itself, which reports 54.6 on AntMaze-Ultra and 90.2 on FrankaKitchen-Partial in the same paper. We use those numbers as the target and also rerun it with the authors' settings for seed-matched comparison. Beating GCPC means exceeding its rerun score on AntMaze-Ultra with non-overlapping confidence intervals over seeds. Matching it within one standard deviation while using only local updates would already count as a positive result for the local-learning question, though not as an improvement in control.
The ablation that isolates the learning rule trains the identical energy and planner with exact gradients from implicit differentiation, that is, ordinary recurrent backpropagation. It uses the same data, the same number of settling steps, and the same wall-clock compute. Any gap between this model and the EP version is the cost of locality.
A second ablation replaces the sum of local transition terms with a single global energy over the whole window. This isolates whether the decomposition, which is the proposed source of stitching, matters.
For the online Meta-World stage, the file's reported numbers come from pixel observations: R-AIF reports 100% success on several sparse tasks, and DreamerV3 reports a relative-stability score of 0.2 against R-AIF's 0.0 on Handle Pull, both in R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models. Our first online stage uses proprioceptive state, so those figures are context, not a fair target. We will rerun DreamerV3 and a contrastive goal-conditioned agent in our state-based sparse setting under a matched environment-step budget.
Why not just…?
Why not just plan by backpropagating through a learned world model? In the limit of small $\beta$ and full convergence, our planner computes the same thing, so a backpropagation planner is the natural upper bound rather than a competitor we expect to beat. This proposal does not claim better control from the local rule. It claims a measurable answer to whether locality survives contact-rich closed-loop control, which matters for neuromorphic and analog hardware where backpropagation cannot run. The implicit-differentiation ablation settles this directly. If the EP version trails it by a wide margin specifically on contact-heavy windows, that is the finding.
Why not just use goal-conditioned behavioral cloning? It needs no settling and no energy. However, the D4RL authors observe that cloning fails when good behavior requires stitching suboptimal segments D4RL: Datasets for Deep Data-Driven Reinforcement Learning, and RvS-G's 26.4 on AntMaze-Ultra reflects this. Our decomposed energy is designed for stitching, and the global-energy ablation tests whether that design, rather than extra test-time compute, is what helps. If RvS-G given equal compute through ensembling matches us, the stitching argument fails.
Why not just use contrastive goal-conditioned RL, which already avoids reward engineering? Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL shows that single-goal contrastive RL solves long-horizon tasks without shaping. In the online stage it is a real threat and may outperform us in sample efficiency. We include it in the Meta-World comparison. Our contribution there is not "reward-free," which it already achieves, but "reward-free and local," and we report the price of that pairing honestly.
Why not just learn a style reward from motion data, as in adversarial motion priors? Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions cuts cost of transport from 1.37–1.65 to 0.93–1.12 in quadruped locomotion. However, it needs retargeted motion capture for a single morphology, which does not exist for Meta-World or Kitchen tasks. It also still requires a task reward term and a global-gradient discriminator, so it addresses a different bottleneck.
Proposed Experiments
The first decisive check is gradient fidelity. On held-out AntMaze and Kitchen windows, we measure the cosine similarity between EP updates and implicit-differentiation gradients as a function of $\beta$, settling steps, and whether the window contains a contact. We predict high alignment in free-space windows and degradation at contacts. If alignment is poor everywhere, the program stops at this negative result.
The second check is offline performance. We report D4RL normalized scores on AntMaze (medium, large, ultra) and FrankaKitchen (partial, mixed) over at least five seeds, with RvS-G, GCPC, and both ablations run under matched budgets. The stitching hypothesis predicts that the advantage of our method over RvS-G grows from medium to ultra mazes, and that the global-energy ablation loses that growth.
The third check is hacking resistance. We count episodes where the planned final state is near $g$ but the executed trajectory is not. We compare this rate with and without the transition energy, using a deliberately weakened $\phi_\theta$ as a stress test.
The fourth check is closed-loop feasibility on Meta-World MT10 tasks with state observations. We report success rate, the variance of success across seeds, and the settling iterations per control step needed to sustain success. That last measure is the quantity that tells a hardware designer whether planning by settling can keep pace with a moving world.
References
- Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation
- Equivalence of Equilibrium Propagation and Recurrent Backpropagation
- Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input
- Sequence Learning Using Equilibrium Propagation
- Active Inference or Control as Inference? A Unifying View
- Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
- Predictive Coding for Locally-Linear Control
- R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models
- Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections
- Equilibrium Propagation for Non-Conservative Systems
- Habitual and Reflective Control in Hierarchical Predictive Coding
- Can the Brain Do Backpropagation? - Exact Implementation of Backpropagation in Predictive Coding Networks
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.
- Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL
- Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions