Critics That Know What the Robot Cannot See: History-State Value Learning for Distributed Manipulation Under Self-Occlusion

When a robot arm closes in on an object, its own gripper often blocks the camera's view at the moment that matters most. Reinforcement learning (RL) systems for manipulation are trained at scale with a split design. Many "actor" processes run simulators and collect experience, and a central "learner" on GPUs updates the network. Because the actors run the simulator, the learner can be handed the true object pose for free, even while the policy itself only sees camera pixels. The standard way to use this is an asymmetric critic: the value estimator, which is needed only during training, reads the true state while the policy reads images. We argue this shortcut is subtly wrong for occlusion. A critic that sees the true object pose cannot tell whether the robot currently knows where the object is. It therefore assigns no value to looking again, and too much value to blind grasps that only succeed when luck is on the robot's side.

We propose training the critic on both the true state and the policy's own memory of what it has seen, a history-state critic. This should be done inside an asynchronous actor-learner system whose stale-policy correction is made consistent with that memory. The prediction is concrete and falsifiable. On occlusion-heavy tasks, history-state critics should produce policies that re-observe, for example by backing off or rotating the wrist to regain sight of the object before committing. State-only critics should not produce these behaviors. The gap should disappear when the occluding gripper is made invisible to the camera.

Core Claim

Unbiased Asymmetric Actor-Critic for Partially Observable Reinforcement Learning proves that a critic conditioned only on state $s$ gives biased policy gradients under partial observability. The reason is that many different observation histories map to the same state, while the policy acts on the history. Conditioning on both history $h$ and state, $V(h, s)$, removes the bias. That result was shown only on small discrete grid worlds with synchronous training. Asymmetric Actor Critic for Image-Based Robot Learning showed large practical gains from a state-only critic on Fetch picking, but in scenes where occlusion was not the central difficulty.

Our claim is that terminal self-occlusion in manipulation is exactly the regime where this theoretical bias becomes a practical failure. The mechanism is specific. A state-only critic values "gripper 1 cm above an object that was just nudged" identically whether or not the policy saw the nudge. As a result, the advantage of an information-gathering action is systematically underestimated.

The decoupled actor-learner setting is where the fix is cheap. Actors already send the learner a few dozen floats of simulator state, which is far less than the image traffic that Scaling Large Vision-Language Model RL Training via Efficient Load Balancing identifies as the dominant bottleneck in multimodal RL. It is also where a new issue appears. The history $h$ is summarized by a recurrent encoder, and under asynchrony the actor computed that summary with stale weights.

Research Program

We build on an IMPALA-style learner (IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures). IMPALA corrects for the gap between the actor's older policy $\mu$ and the learner's current policy $\pi$ using clipped importance ratios, a method called V-trace. We use a Gaussian policy with a recurrent image encoder that reads wrist and fixed camera images. Actors are run in a SURREAL-like process layout (SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark), and each actor streams images, actions, behavior log-probabilities, and privileged simulator state.

The learner re-encodes each trajectory segment with current weights after a short burn-in. This yields history summaries $\hat h_t$ that are consistent with $\pi$. The critic is trained on V-trace targets:

$$v_t = V_\phi(\hat h_t, s_t) + \sum_{k \ge t} \gamma^{k-t} \Big(\prod_{i=t}^{k-1} c_i\Big) \rho_k \big(r_k + \gamma V_\phi(\hat h_{k+1}, s_{k+1}) - V_\phi(\hat h_k, s_k)\big)$$

Here $V_\phi$ is the critic with parameters $\phi$, $r_k$ is the reward, $\gamma$ the discount, and $\rho_k = \min(\bar\rho, \pi(a_k|\hat h_k)/\mu(a_k|h^{\mu}_k))$, with $c_i$ defined the same way using a separate clip $\bar c$. The term $h^{\mu}_k$ is the stale summary the actor actually used.

The one design question we study beyond the critic input is the mismatch between the history in the numerator and the history in the denominator. This mismatch is exactly the tension the research file flags as unstudied. We compare three treatments. The first stores the actor's summaries. The second re-encodes with burn-in. The third uses SEED-style central inference (SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference), which makes $h^{\mu}$ nearly equal to $\hat h$ at the cost of streaming images.

The analysis instrument is an occlusion mask. It is computed from simulator segmentation and gives, at each step, the fraction of target-object pixels hidden by the robot. Using this mask, we measure three things. First, the critic's value error stratified by occlusion level. Second, the advantage the critic assigns to re-observation actions. Third, how often trained policies regain visibility before closing the gripper.

Baselines

The naive reference is a symmetric recurrent actor-critic in the same distributed system, where both policy and critic see only images. This approximates the pixel-based SURREAL-PPO setting. In that setting, SURREAL reports Bin Picking and Nut-and-peg Assembly unsolved even at 64 actors, though no numeric score is reported on Lacuna for those tasks. Any method that does not clearly beat this reference on occlusion variants is not worth pursuing.

The strongest competitor is the state-only asymmetric critic of Asymmetric Actor Critic for Image-Based Robot Learning, ported to the same V-trace learner. On Fetch Pick that paper reports near-perfect success versus under 40% for its symmetric baseline, without an exact figure. We found no reported result on Lacuna for a state-only asymmetric critic on occlusion-heavy manipulation. It is the strongest competitor because it uses the same privileged information we do, and it differs only in ignoring history. If history does not matter in practice, this baseline will match us.

For the active-viewing comparison on robosuite NutAssemblySquare, the external reference is Active Vision Reinforcement Learning under Limited Visual Observability. It reports a score of 108.6 for SUGARL, 102.9 for a hand-coded fixed view, and 8.42 for a single joint policy. That method assumes a camera independent of the arm, so we report it as a reference only for the decoupled-camera variant.

All learned baselines share the encoder, actor count, environment frames, and hyperparameter search budget. We tune each method over the same grid of learning rate, entropy cost, and clip thresholds with three seeds per configuration. Final comparisons use ten seeds.

The key ablation swaps only the critic input among $V(s)$, $V(\hat h)$, and $V(\hat h, s)$, with everything else fixed. A second ablation varies the history-consistency treatment described above. Beating the strongest baseline means a statistically significant success-rate gain on occlusion variants together with no loss on the visibility-control variant. A gain that appears equally in the control would mean we found a capacity effect, not the bias effect.

Why not just…?

Why not just use the state-only asymmetric critic, since it already works? This is the real threat. In many manipulation tasks the true state may make history nearly irrelevant to value, so the bias could be negligible. The theory in Unbiased Asymmetric Actor-Critic shows the bias appears only when histories mapping to the same state lead to different behavior. Our task variants are built so that this either happens or does not, which lets the experiment settle the question. If $V(s)$ matches $V(\hat h, s)$ on the perturbed-while-occluded variant, the core claim is wrong and we will say so.

Why not just add cameras or re-render the scene from virtual views? RVT: Robotic View Transformer for 3D Object Manipulation fuses calibrated multi-camera point clouds into virtual views and reaches 62.9% average success on 18 RLBench tasks. If enough cameras exist, occlusion largely disappears and so does our problem. We therefore treat a privileged multi-view policy as an upper reference, not a competitor. The relevant regime is the one where camera count, calibration, or thin and reflective objects limit coverage, which RVT itself notes are weaknesses. Static rigs also underperform active viewing in Active Vision Might Be All You Need: 30% versus 52% on Thread Needle. That is consistent with occlusion persisting under fixed views.

Why not just remove policy lag with central inference and ignore the history mismatch? SEED RL largely removes behavior staleness. It is included as one arm of the history-consistency ablation, so this objection is tested directly. It pays for that by streaming every image to the learner, which is the bandwidth cost highlighted in Scaling Large Vision-Language Model RL Training via Efficient Load Balancing. If re-encoding with burn-in matches SEED's success at lower actor-to-learner traffic, the cheaper design wins. If it does not, we recommend SEED.

Why not just give the policy a separate camera controller? Modular eye-hand designs (Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions, 71.0% versus 55.8% static under heavy clutter) need curricula and an independent camera. With a wrist camera, looking requires moving the hand, and that is precisely the behavior a state-only critic undervalues. The critic fix is complementary to active vision rather than an alternative to it.

Proposed Experiments

We use robosuite (robosuite: A Modular Simulation Framework and Benchmark for Robot Learning) with Lift, Pick-and-Place, and NutAssemblySquare. Policies see a wrist camera plus one fixed camera.

Each task comes in three variants. In the standard variant, the task is unchanged. In the perturbed-while-occluded variant, the object receives a small random shift only when the occlusion mask exceeds a threshold, which makes what the robot last saw decisive. In the visibility control variant, the robot's gripper and links are rendered invisible to the cameras but remain physically present. The control matters because it holds dynamics identical while removing the information loss. Our theory predicts that the advantage of $V(\hat h, s)$ over $V(s)$ appears in the perturbed variant, is smaller in the standard variant, and vanishes in the control.

As a secondary check of generality, we run the FetchPush occluder suite from Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions.

Metrics are success rate and return against environment frames and wall-clock time. We also track the behavioral and diagnostic measures from the research program: fraction of episodes that regain object visibility before gripper closure, critic value error binned by occlusion level, and estimated advantage of scripted re-observation actions.

To test interaction with asynchrony, we sweep artificial policy lag from zero to many learner updates. We expect the stored-summary treatment to degrade fastest under occlusion, because the stale encoder's memory matters most when current pixels are uninformative.

The decisive checks are three. First, a success gap between history-state and state-only critics that grows with occlusion and disappears in the visibility control. Second, a matching gap in re-observation frequency, so that the mechanism, not just the score, is shown. Third, a demonstration that re-encoded histories keep V-trace stable across the lag sweep at a fraction of central-inference bandwidth. Failure of the first check falsifies the proposal outright.