Reinforcement learning needs a reward function, and for physical control someone usually has to write it by hand. For a walking robot this means trading off forward speed, posture, foot clearance, energy and jerk penalties, and the weights are fragile. Methods that learn rewards from video avoid this, but nearly all of them score a frame by how close it looks to a goal frame. That fails in two ways. Tasks like running or balancing have no goal frame. And a reward computed from a single snapshot can be shifted by anything in the image, including background, color and lighting. We propose to reward the agent's motion instead of its position. A learned encoder maps the change between consecutive frames, $z_t - z_{t-1}$, to a compact motion code. The reward is high when a short window of the agent's motion codes matches some window of motion codes from unlabeled, action-free demonstration videos.
The bet rests on what subtraction does to an image. When you subtract consecutive frame embeddings, anything that stays fixed cancels, including scenery, color schemes and the agent's appearance. What remains is velocity and direction. Policy architectures that use such differences have shown large gains, but they still train on hand-written rewards. Video-derived rewards have removed hand-written rewards, but they still encode static frames. This note puts differencing inside the reward model. It then tests three claims: that this yields usable rewards for goal-free locomotion, that the rewards hold up better under appearance shift, and that they produce smooth behavior without a hand-tuned jerk penalty.
Core Claim
Our claim is about non-terminating continuous-control tasks where only video of competent behavior is available. On these tasks, a reward defined as windowed nearest-neighbor similarity in a learned space of frame-difference codes will do three things. First, it will train agents to a substantial fraction of the ground-truth return reached with the hand-designed reward. Second, it will lose less performance than static-frame reward models when colors and static backgrounds change between the demonstration videos and the agent's environment. Third, it will produce lower action jitter than the hand-designed reward does, because demonstrations are smooth and jerky motion produces motion codes that have no match.
What is new is the place where differencing sits and the form of the reward. In Reinforcement Learning with Latent Flow, Flare encodes each frame and feeds the policy the latent difference. In Visual Reinforcement Learning with Residual Action, ResAct subtracts the feature maps of consecutive frames to predict changes in action. Both supply our encoding method, and both depend entirely on DeepMind Control and CARLA rewards written by hand. Flare also reports jittery policies, which is part of what motivates our smoothness claim.
CoMo subtracts ViT features of current and future frames and passes the result through a 128-dimensional bottleneck. It does this so that inferred actions do not memorize the background, and it uses the result for behavioral cloning rather than as a reward. We borrow its difference-plus-bottleneck design for a different purpose.
XIRL aligns demonstration videos across different bodies with temporal cycle-consistency. Temporal cycle-consistency, or TCC, trains embeddings so that matching moments in different videos map to each other. VIRL builds object graphs from each frame and aligns them the same way. Both reward negative distance to a goal frame, encode frames independently, and cannot reward cyclic tasks. These papers motivate the gap. We reuse TCC, but we apply it to sequences of differences rather than sequences of frames.
Research Program
The reward model has three parts. A frame encoder $f$ is either a frozen pretrained ViT or a small CNN trained jointly. A difference head $g$ maps $\Delta z_t = f(o_t) - f(o_{t-1})$ through a low-dimensional bottleneck to a motion code $m_t$. The training objective applies TCC across demonstration videos to windows $M_t = (m_{t-k+1}, \dots, m_t)$, so that the same phase of a gait in two differently rendered videos lands at the same point.
Given demonstration windows $\{M^d_j\}$, the reward is
$$r_t = -\min_j \lVert M_t - M^d_j \rVert_2 ,$$
where $M_t$ is the agent's most recent window of $k$ motion codes.
This form makes three properties concrete:
- Standing still is penalized. A stationary agent produces near-zero codes that match no running window.
- Direction matters. Codes carry the sign of the change, so moving backward does not score like moving forward. Distance to a goal frame cannot tell these apart, and Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping showed that symmetric distances fail on directed dynamics.
- Windowing limits one kind of reward hacking. Matching windows rather than single steps keeps the agent from replaying one locally matching twitch.
We will analyze where the remaining risk lies. One case is an agent that matches a gait window while making no forward progress. In DeepMind Control, a tracking camera makes the floor texture move across the image, so progress shows up in the differences, but this is itself a scene cue and must be checked. A second case is dynamic backgrounds, which do not cancel under subtraction. Here we expect partial failure and test whether the bottleneck limits it.
Demonstrations come from expert policies trained with the ground-truth reward, recorded as action-free video. Crucially, they are rendered under visual conditions different from the agent's. For cross-body transfer we use X-MAGICAL's stick and gripper embodiments. In every experiment the learner never sees the ground-truth reward, which is used only for evaluation.
Baselines
The naive reference points come from the same RL agent (RAD-style augmentation with a Flare encoder) in two settings. The upper reference trains on the hand-designed DeepMind Control reward. The floor trains on a zero-information reward, meaning a random policy. For this suite, the file reports only relative results: Flare is 1.4x RAD at 1M steps and 1.3x at 500k steps, and ResAct is best in sample efficiency on 5 of 6 tasks. No absolute DMC returns were found on Lacuna. We therefore rerun the manual-reward agent ourselves and report learned-reward returns as a fraction of it.
The strongest existing method is XIRL's TCC goal-distance reward. On X-MAGICAL we run it as published. On cyclic tasks we adapt it by replacing distance-to-goal with distance to the nearest demonstration frame, which is its most natural extension. On the real xArm pushing task, VIRL reports 55% success for VIRL, against 5% for XIRL and 20% for hand-designed rewards. These figures show that frame-based learned rewards can beat manual ones but also vary widely. For X-MAGICAL itself, the file gives only a qualitative statement that XIRL reaches high success in about half the steps of TCN. No numeric X-MAGICAL result was found on Lacuna.
A frozen-foundation-feature baseline rewards nearest-neighbor distance in frozen DINOv2 frame embeddings.
An adversarial imitation-from-observation baseline trains a GAIL-style discriminator on frame pairs $(o_{t-1}, o_t)$.
The key ablation keeps our objective, windowing and bottleneck but feeds the encoder stacked frames $[o_{t-1}, o_t]$ instead of their difference. This isolates subtraction as the ingredient. Further ablations remove the bottleneck and set the window length $k$ to 1.
Fairness. All methods share the same demonstration videos, the same RL algorithm and step budget (500k and 1M environment steps), the same encoder backbone where applicable, and an equal hyperparameter-search budget per method. We use at least five seeds.
What counts as success. Beating the baselines means two things. Under appearance shift, our method must achieve higher ground-truth return than every learned-reward baseline, with non-overlapping seed confidence intervals. The stacked-frame ablation must lose measurably, especially under color and background shift.
Why not just…?
Why not just stack frames and let the reward encoder learn motion itself? Flare found that stacking underperformed explicit latent differencing in policies: 1668 versus 900 on Montezuma's Revenge, and failure to recover velocity in state-based ablations (Reinforcement Learning with Latent Flow). However, that evidence comes from policies, not reward models. The stacked-frame ablation answers the question directly. If it matches our method under shift, subtraction is not the active ingredient, and we would report that.
Why not just use frozen DINOv2 or CLIP distances? CoMo reports that even pretrained ViT features invite background shortcuts unless they are differenced. The Distracting Control Suite reports that a background video blended at under 10% opacity roughly halves the scores of top methods. Both suggest frozen features alone will not survive appearance shift. The frozen-feature baseline is run under exactly those shifts to confirm or refute this.
Why not just use adversarial imitation from observation? This is the real threat. A discriminator on frame pairs can also learn motion. It has two weaknesses for our setting. It must be retrained against every new agent, so it is not a fixed reward that transfers. And in MAGICAL, GAIL fell from 0.72 on the demonstration setting to 0.22 under layout shift and 0.27 under color shift. If it matches us in the unshifted setting, our claim narrows to robustness and cross-body transfer. We say so in advance.
Why not keep the hand-written reward and add a smoothness penalty? That reintroduces exactly the weight tuning that Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion found fragile. ResAct gets smoothness from a change in architecture, not from a reward. We compare jitter with ResAct's Mean Action Distance metric, which measures how much consecutive actions differ, against the manual reward both with and without a tuned penalty.
Proposed Experiments
DeepMind Control locomotion. On Walker Walk and Run, Cheetah Run and Hopper Hop, agents learn only from our reward and from each baseline reward. We report ground-truth return at 500k and 1M steps as a fraction of the manual-reward agent's return, together with Mean Action Distance.
Robustness to appearance shift. On the Distracting Control Suite, demonstrations are rendered clean and the agent acts under color, camera-pose and background distractions. Intensities sweep from 0 to 1, in both static and dynamic modes. We plot how return degrades for each reward model. Our prediction is that differencing helps most for static color and background shifts and least for dynamic DAVIS video backgrounds. Finding the opposite would undercut the mechanism.
Goal-directed sanity check and cross-body transfer. On X-MAGICAL, motion codes are trained on three embodiments and the reward is evaluated on the fourth. This checks that rewarding motion does not cost goal-directed performance relative to XIRL.
Reward diagnostics. We report the correlation between the learned reward and the hidden ground-truth reward along rollouts. We also hand-inspect high-reward behaviors that score low on ground truth, which is how we look for in-place oscillation hacks.
Decisive checks. The method is supported only if three things hold under shift: it beats the stacked-frame ablation, it beats the frozen-feature and adversarial baselines, and it shows lower jitter than the manual reward without a penalty. Losing the first comparison means differencing is not the cause. Losing the last means the smoothness claim fails.
References
- Reinforcement Learning with Latent Flow
- Visual Reinforcement Learning with Residual Action
- CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
- XIRL: Cross-embodiment Inverse Reinforcement Learning
- VIRL: Self-Supervised Visual Graph Inverse Reinforcement Learning
- Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
- The Distracting Control Suite - A Challenging Benchmark for Reinforcement Learning from Pixels
- The MAGICAL Benchmark for Robust Imitation
- Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion