Robots learning multi-step chores by trial and error face a basic problem. They are usually told only at the very end whether they succeeded, which leaves them no signal about which of hundreds of earlier actions mattered. Recent work fills this gap with "progress critics": learned models that watch camera frames and score how far along the task looks. These critics share two weaknesses. They judge progress from what the camera sees right now, so a drawer hidden behind the robot's own arm looks the same as a drawer that was never opened. They also assume progress rises steadily over time, so stepping back to clear an obstacle reads as going backwards. This proposal computes the reward from a persistent 3D memory of the scene instead. The memory is a running record of where each object is and what state it is in, and it keeps that record when objects leave the camera's view. Progress becomes a count of how many goal conditions the remembered scene currently satisfies, checked by small executable tests. Elapsed time plays no role.
In two sentences: we turn the robot's spatial memory into the source of its dense reward, so progress is measured by changes in the remembered world state rather than by camera pixels or the clock. Rewards of this kind should keep working under occlusion and should not penalize necessary detours, which are the two places where current critics fail.
Core Claim
Programmatic checks on a persistent object-state memory give a denser and more reliable reinforcement-learning reward for long, occlusion-heavy manipulation than frame-based progress critics. The gain should grow with how often task-relevant objects are out of view.
The closest existing pieces each cover one part of this. Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action (SOMA) builds the kind of memory we need. It lifts detections, DINOv3 features and VGGT geometry into a 3D map that is refreshed with an exponential moving average. However, it uses the map only as policy input, under pure imitation learning, and it does not track articulation state such as how far a drawer is open. Active Exploring like a Pigeon supplies the reward technique. It writes spatial relations as executable Python assertions over a coordinate map and uses them as verifiable step rewards, and removing them cost 10.2% accuracy on MindCube. But that map is static and the task is question answering, not physical control. Three progress critics motivate the proposal by showing the gap. VLAC uses pairwise frame comparison, RFTF uses temporal contrastive ranking, and SARM uses stage-plus-progress heads. All three read egocentric frames with no persistent spatial anchor, the first two assume monotonic progress in time, and SARM needs hand-labelled stage boundaries. Our proposal differs from all of them in where progress is read: from a world-state record that survives occlusion and is indifferent to time.
Research Program
The system has three parts. The first is a state-augmented spatial memory. It extends SOMA-style lifting so that each object entry stores, beyond its pose and appearance, a small set of continuous state variables, together with a confidence value that decays while the object is unobserved. Examples of state variables are a drawer's opening fraction, estimated from the displacement of a handle keypoint along the cabinet's sliding axis, a switch or light state, and contact or containment relations.
The second part is a goal-assertion compiler. It turns each task instruction into a small set of executable checks over the memory, in the style of the assertion codes from the Pigeon paper. Examples are "the drawer opening fraction exceeds 0.8" and "block_red lies inside the drawer volume". On CALVIN's 34 templated tasks these checks can be written once per task family. For free-form instructions, a language model drafts them and a human reviews them, and we report how much review effort this takes.
The third part is the reward. We use the standard potential-based shaping form, which leaves the optimal policy unchanged:
$$\Phi(m_t)=\sum_k w_k\, c_k(m_t)\, s_k(m_t),\qquad r_t = r^{\text{sparse}}_t + \gamma\,\Phi(m_{t+1})-\Phi(m_t).$$
Here $m_t$ is the memory at step $t$, $s_k\in[0,1]$ is a soft truth value for assertion $k$, $c_k$ is the confidence of the objects that assertion reads, $w_k$ is the assertion's weight, and $\gamma$ is the discount. The reward has two useful properties. When the robot looks away, the memory freezes rather than resets, so $\Phi$ does not drop and the detour goes unpunished. When confidence in an object falls below a threshold, its assertions stop contributing until the object is seen again. The system then emits a re-observe sub-goal, which directly addresses the stale-memory problem SOMA leaves open.
This reward drives PPO fine-tuning of a pretrained vision-language-action policy, meaning a model that maps images and instructions to robot actions. To keep the comparison clean we use the same Seer-Large policy, advantage estimation and sample balancing that RFTF used, and we change only the reward. Cyclic tasks such as wiping are explicitly out of scope, since counting satisfied conditions does not capture repetition. We report this as a limitation rather than claim it.
Baselines
The naive reference point is PPO with the environment's sparse success reward alone. On CALVIN with training on environments A–C and testing on the unseen environment D, RFTF reports this at 4.225 average completed-task length with Seer-Large, in RFTF.
The strongest competitor on the main benchmark is RFTF itself. It is a learned temporal value model trained on expert video, converted into dense rewards for PPO, and it reports 4.296 average length on unseen environment D and 4.301 with few-shot adaptation. It is the strongest competitor because it is the only cited method that uses dense rewards to fine-tune a large manipulation policy on a multi-stage benchmark. We will rerun it with its published recipe on the same policy checkpoint, the same number of environment interactions and the same hyperparameter-search budget as ours. On the standard CALVIN split, matching it would already be acceptable, because the cameras there rarely lose sight of objects. Beating it means a statistically reliable gain on our occlusion variant, where the gap should be widest.
The second competitor is VLAC, a pairwise frame-comparison critic trained on 40 million instances. It lifted a real robot from 30% to 90% success in 200 episodes in VLAC. No reported result was found on Lacuna for VLAC on CALVIN or RoboCasa. We will run the released model zero-shot as a reward source under the same PPO budget.
SARM serves as a stage-aware reward baseline. We give it the stage boundaries our assertions imply, which favours it on annotation cost.
A privileged-state oracle computes the same assertions from the simulator's true object states. It is an upper bound, not a competitor.
Two ablations isolate the key ingredient. The first, the persistence ablation, evaluates the identical assertions on current-frame detections only, with no memory. If it matches the full method, persistence is not doing the work. The second removes confidence gating. It tests whether stale entries cause reward hacking or false progress.
Why not just…?
Why not just compute rewards from the simulator's true state? In simulation that is strictly better, and it is our oracle. It does not exist on real robots, however, and it hides the question that matters: whether a reward built from the robot's own perception is good enough to train on. The useful quantity is how much of the oracle-over-sparse gain our perceived-memory reward recovers. We also measure how often its assertions disagree with ground truth, and whether the disagreements concentrate under occlusion.
Why not just give a frame critic a longer history? A critic that reads many past frames could in principle infer occluded state. ReLIC shows implicit memory over 64k steps can work for navigation, but its in-context memory use emerged only after roughly 150 million training steps and needed special attention sinks. This is a real threat for short tasks. We test it directly by feeding VLAC a multi-frame window and by training an RFTF-style critic on clip inputs. If either matches our method on the occlusion variant, the explicit memory is not justified.
Why not just add SOMA's memory to the policy and keep RFTF's reward? This tests whether the benefit comes from memory as input or memory as reward. We include it as a cross-condition: memory-input policy with RFTF reward, against memory-input policy with memory reward. SOMA's 28.3% versus 18.5% for GR00T-N1.5 on its out-of-view benchmark shows that memory helps as input. Our claim fails if this cross-condition closes the gap.
Proposed Experiments
We start on CALVIN's unseen-environment split as a sanity check, where we expect parity with RFTF's 4.296. The decisive test is a CALVIN occlusion variant we construct: only a wrist camera, plus a head-camera motion schedule that takes the drawer, slider and blocks out of view between sub-tasks. We sweep the fraction of out-of-view steps and plot average completed-task length against it for every reward source. The prediction is that frame critics degrade with occlusion while our reward and the persistence ablation separate.
For transfer beyond CALVIN, we repeat on RoboCasa, where SOMA reports 49.3% as an imitation-only reference, and on a small set of long RLBench tasks with default sparse rewards.
We run three diagnostics. The first measures non-monotonicity: we inject forced detours into rollouts and record each reward source's response, where ours should stay flat and time-based critics should go negative. The second measures reward fidelity: the per-step agreement between memory assertions and simulator truth. The third is a hacking audit: rollouts where success comes from the shaped reward but the environment reports failure.
If budget allows, a final real-robot check on the out-of-view setup from SOMA would compare sample efficiency against VLAC's 200-episode curve.
References
- Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
- Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- RFTF: Reinforcement Fine-tuning for Vision-language-action Models with Temporal Feedback
- SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
- ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI