Detecting Reward Shifts in Sparse-Reward Visual Control by Watching Where Progress and Success Disagree

Robots that learn manipulation from camera images usually receive only a yes-or-no success signal at the end of each attempt. Current methods get around this by turning a handful of demonstrations into a dense progress signal. They measure how close the robot's current image is to the demonstration images in the feature space of a frozen, pre-trained vision network. This works well when the task stays fixed. It fails silently when the task changes: the target moves, the success criterion tightens, or the lighting changes. The progress signal keeps saying "almost done" while the real reward stays at zero, and the agent keeps optimizing a stale objective. Detecting such changes from the sparse success signal alone is slow, because a drop in success rate looks the same as a policy that is simply still bad.

We propose a cheap detector for this situation. It watches for disagreement between the dense progress proxy and the sparse outcome. For attempts where the proxy says the robot reached the demonstrated end state, success should be nearly guaranteed. A sustained drop in that conditional success rate is strong, fast evidence that the reward changed. A shift in the proxy values themselves, with no drop in conditional success, points instead to a change in appearance. The two cases call for different responses. In the first, the demonstrations are no longer valid and the agent should re-anchor its shaping on its own new successes. In the second, only the feature normalization needs recalibrating and the policy should be left alone. Both tests run on scalars, one per episode, so they cost almost nothing next to the policy and encoder.

Core Claim

In sparse-reward visual control with demonstration-derived shaping, the right statistic for detecting reward non-stationarity is not the raw success rate. It is the success rate conditioned on the proxy reporting completion. Conditioning removes most of the variance due to policy quality. A standard change-point test (a sequential likelihood-ratio test on a Bernoulli stream) on this conditioned stream should therefore detect goal and criterion shifts in fewer episodes than a test on unconditioned success. A second test on the proxy's own distribution separates visual shifts from reward shifts. Pairing detection with a targeted response gives faster post-shift recovery than existing methods, at a small fraction of their compute. The two responses are re-anchoring shaping on hindsight successes and recalibrating without resetting the policy.

Formally, let $p_k$ be the proxy's end-of-episode progress in episode $k$, $\tau$ a completion threshold, and $y_k \in \{0,1\}$ the sparse outcome. Over the gated episodes $\{k : p_k \ge \tau\}$ we run a generalized likelihood ratio test for a single change in the success probability $q$:

$$ \Lambda_n = \max_{1 \le s < n} \Big[ s \,\mathrm{kl}(\hat q_{1:s}, \hat q_{1:n}) + (n-s)\,\mathrm{kl}(\hat q_{s+1:n}, \hat q_{1:n}) \Big], $$

Here $n$ is the number of gated episodes so far, $\hat q_{a:b}$ is the empirical success rate over gated episodes $a$ to $b$, and $\mathrm{kl}$ is the Bernoulli Kullback–Leibler divergence. We declare a change when $\Lambda_n$ exceeds a threshold set for a target false-alarm rate. Detection delay for such tests scales roughly as the threshold divided by $\mathrm{kl}(q_{\text{before}}, q_{\text{after}})$. Gating raises $q_{\text{before}}$ from the policy's overall success rate toward one. That enlarges the divergence and shortens the delay, and this is the quantitative expectation we test.

Positioning Against Prior Work

The proposal reuses the shaping machinery of LaNE. LaNE turns one to five demonstrations into dense exploration rewards using a frozen DINOv2 encoder and a latent distance that is invariant to augmentation. It trains a Franka Panda from scratch in under an hour, but it explicitly assumes fixed goals and rewards. We keep its latent space as the proxy and add what it lacks: a way to notice when the demonstrations have become wrong.

Two works motivate the problem and constrain the design. QUEST names the "moving target problem": a learned dense reward and a policy shift together and destabilize learning. QUEST handles this with an ensemble of five visual dynamics models plus curiosity and discriminator networks. That ensemble addresses self-induced non-stationarity, not external task change, and it is expensive. VLAC shows that pair-wise progress critics built on 2B–8B vision-language models raise real-robot success from 30% to 90% in 200 episodes. It also documents that such critics assume monotone progress and lack any change detection. Both papers say that dense visual proxies are powerful and brittle. We treat the proxy's brittleness as a signal instead of a liability.

The non-stationarity literature supplies the statistical tools, but only for other settings. DAL wraps any stationary bandit algorithm with GLR and Shiryaev–Roberts change detection plus forced exploration over a minimal covering set. Testing stationarity and change point detection in RL applies CUSUM tests to fitted Q-functions on offline, low-dimensional data. Neither can be applied directly to pixel-based actor-critics, because the objects they test do not exist there in low-dimensional form. Our contribution is to choose a scalar stream whose statistics are both cheap and informative. Because it is scalar, the DAL-style "detect, then reset" recipe carries over to visual control.

FANS-RL is the closest competitor. It infers separate latent factors for dynamics and reward changes with a causal variational model, and it works from pixels on Sawyer-Peg. However, it relies on dense rewards and heavy variational inference. Pausing Policy Learning shows that holding policy weights fixed at a sub-1.0 update frequency beats continual updating under shifts, but only with dense rewards, low-dimensional states, and a fixed schedule. Our visual-shift branch is a detection-triggered version of that pausing idea: when appearance changes but reward does not, we hold the policy while recalibrating.

Research Program

We build three components on top of a LaNE-style agent: a sample-efficient off-policy actor-critic, a frozen DINOv2 encoder, a learned dynamics-aware latent, and demonstration-anchored shaping.

The first component is the reward-shift test described above, run on gated episodes.

The second is an appearance-shift test. It is a CUSUM on the mean and variance of the proxy's latent distances along the demonstration-matched segment of each episode, which tracks whether the feature statistics have moved.

The third is a response policy for each alarm. A reward-shift alarm sets the demonstration shaping weight to zero, adds a short burst of forced exploration, and promotes the agent's first new sparse successes to become the new anchor trajectories. This is hindsight re-anchoring, in the spirit of relabeling achieved outcomes as goals. A visual-shift alarm without a reward-shift alarm freezes policy updates. It then refits the latent normalization and the small dynamics-aware projection on fresh frames, and resumes.

The analysis has three parts. First, we measure detection delay against false-alarm rate as a function of the policy's pre-shift success rate. This shows empirically where gating helps and where it does not: gating should matter most when overall success is moderate, and little when success is already near one. Second, we test whether the proxy threshold $\tau$ can be set from the demonstrations alone, as the proxy value the demonstrations themselves attain. Third, we characterize failure cases, especially shifts the proxy cannot see. An example is a success criterion that changes without changing any visible end state, such as a required hold time. For these, gating gives no advantage and the method should fall back to the performance of an unconditioned test.

Baselines

The naive reference is the unmodified LaNE agent trained straight through the shift, with continual updates and no detection. It shows how badly stale shaping hurts. We run it from the official code (PhilipZRH/LaNE) with its published hyperparameters. Any recovery we report must beat it in episodes-to-recover after the shift. No reported result was found on Lacuna for LaNE's simulator success rates, so we will reproduce it ourselves.

The strongest existing method for sparse-reward visual manipulation from few demonstrations is QUEST. The file reports that QUEST beats Demo3 and MoDem2 by 17% on average across 16 visual tasks, with gains up to 60% on high-precision tasks. It also reports 62% versus 32% average real-world sim-to-real success. No absolute simulator success numbers are given, and nothing is reported under reward shifts. We will run QUEST on the same shifted tasks with the same demonstrations and the same environment-step budget. We will report both success and GPU-hours, because QUEST's ensemble uncertainty may itself partially react to shifts. Beating it means matching or exceeding post-shift cumulative success at a fraction of its memory and wall-clock cost.

For non-stationarity handling, FANS-RL is the strongest pixel-based competitor. Its reported Sawyer-Peg-from-pixels average return is 18.01, versus 12.45 for ZeUS and 6.38 for Meld, under dense rewards. No reported result was found on Lacuna for FANS-RL under sparse rewards. We will first reproduce the dense-reward setting as a sanity check against the 18.01 figure. We will then run both methods on a sparse-success version of the same task, with matched environment steps and a matched hyperparameter search of equal size.

A fixed-schedule pausing baseline in the style of Pausing Policy Learning tests whether detection matters at all, or whether holding the policy on a timer is enough. We sweep its update fraction with the same tuning budget we give our thresholds.

Three ablations isolate the key ingredient. One replaces the gated test with the same GLR test on unconditioned success. One removes the visual-shift branch, so every alarm triggers a full re-anchor. One replaces hindsight re-anchoring with a plain reset of the shaping weight. The first ablation is the decisive one.

Why not just…?

Why not just run a change-point test on the raw episode success rate? This is the most serious alternative, and it is the first ablation. When the pre-shift policy already succeeds almost always, conditioning adds little, and the raw test should do about as well. We expect the gated test to win in the regime that matters in practice: tasks where success is still moderate, because learning is ongoing or the task is high-precision. There, a raw drop from 50% to 0% is statistically much weaker evidence than a gated drop from 95% to 0%. If the delay curves show no gap at moderate success rates, the core claim is wrong. The contribution would then shrink to the response policy and the visual/reward disambiguation.

Why not just keep training continuously and let the agent re-adapt? The naive baseline measures this directly. There is also a mechanistic reason to expect it to fail: the demonstration shaping actively pulls the agent back toward the old goal, so continual training fights the new sparse signal instead of following it. The pausing results in Pausing Policy Learning further show that continual updating is suboptimal even when rewards are dense.

Why not just use QUEST's ensemble uncertainty or FANS-RL's latent factors as the shift detector? Both might react to shifts, and we will measure whether they do. However, both answer a different question at much higher cost. Ensemble disagreement reflects novelty in the dynamics, and a goal move need not change the dynamics at all. FANS-RL's reward factor must be inferred from reward signals that are zero almost everywhere in the sparse setting. If either one detects reward shifts as fast as the gated test, our method's advantage reduces to compute cost, and we would report it that way.

Proposed Experiments

We build shifted variants of Robosuite (drawer, lift, peg insertion), Meta-World, and ManiSkill3 tasks. Each variant uses 5 demonstrations and one of three shift types introduced partway through training: an abrupt target relocation, a tightened success tolerance, or a lighting and texture change. Tasks with drifting targets test the known weakness of change-point methods under continuous drift. We also add sparse Sawyer-Peg for comparison with FANS-RL.

We report four groups of metrics: detection delay in episodes and false alarms per 1,000 episodes; episodes to recover 80% of pre-shift success; cumulative success over the full run; and peak GPU memory and wall-clock time per million steps.

Three checks decide the outcome. First, the gated test must show shorter detection delay than the raw-success test at pre-shift success rates between roughly 30% and 70%, consistent with the divergence argument above. Second, on lighting-only shifts, the visual-shift branch must avoid the loss in success that an unconditional reset causes. Third, the whole system must match or exceed QUEST's post-shift success while using well under its memory footprint. A small real-robot run on a Franka arm with a mid-session target relocation will test whether the delay gains hold outside simulation.