A robot that must learn a manipulation task by trial and error, with a reward only when the task is finished, needs a reason to explore in the meantime. The usual reason is curiosity: an extra reward paid whenever the robot's own model of the world fails to predict what it will see next. Curiosity has a known weakness, the "noisy TV problem." A source of unpredictable change, such as a screen showing static, keeps prediction error high forever, so the agent learns to stare at it instead of exploring. We argue that camera-based manipulation contains a noisy TV the robot builds for itself. When the gripper closes in on an object, the arm blocks the camera. When it moves away, the scene reappears in a state the model could not have predicted, because the relevant change happened out of view. The robot is paid for occluding and revealing, and nothing in that payment reflects anything learnable about the task.
The proposal is to score curiosity against a better-informed predictor instead of the robot's own. During training in simulation, a second model sees every camera, remembers recent history, and knows where the robot's body is. It predicts the same next-step target as the policy's single-camera model. Prediction error that disappears once the robot "could have seen" comes from occlusion. It is not rewarded as novelty, and it becomes a signal for where to point a camera. Error that remains even for the fully informed model is either true novelty or irreducible noise. Learning progress, meaning whether the error falls with training, tells those two apart. The result is a curiosity signal meant to ignore both the background TV and the self-made occlusion TV while still driving exploration toward the sparse goal.
Core Claim
Our claim is that, in pixel-based sparse-reward manipulation, a measurable share of curiosity reward is produced by the robot occluding and revealing its own workspace. Two remedies from the noisy TV literature fail to remove it. Inverse-dynamics features fail because the occluder is the agent's own action-controlled arm. Learning-progress monitoring fails because reveal errors mix a learnable part (contact dynamics) with an unlearnable part (information the camera never had). Separating "hard to predict" from "hard to see" by comparing two predictors that share a target but differ in inputs should restore efficient exploration. The intended scope is honest: the effect should matter most where hidden state actually varies, as in contact, pushing, and object pose jitter. It may be negligible in tasks where nothing changes while the object is hidden. The first experiment exists to find out which case holds.
Position Relative to Prior Work
The method comes from the noisy TV literature. Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring (LPM) rewards the drop in a dynamics model's error over training rather than the error itself, so unlearnable transitions earn nothing. It was tested on Atari, MountainCar and a 3D maze, never on manipulation. The file also records its known failure: when the world model cannot learn the dynamics, the agent gets bored too early. We reuse learning progress, but compute it on the fully informed predictor. The intent is that occlusion-induced error, which a single-view model can never reduce, does not trigger that early boredom or leak reward. BAMDP Shaping shows that intrinsic bonuses resist hacking only when they track the true value of information, with bounded, monotone potentials. It was shown only in discrete toy domains. It constrains our design: our reward must go to zero on stationary noise, and we check this directly.
The occlusion literature motivates the problem. Active Vision Might Be All You Need (AV-ALOHA) adds a camera arm and raises needle-threading success from 30% to 52%. It relies on teleoperated demonstrations and reports that the larger 21-joint action space hurts simpler tasks. VFAS-Grasp documents the terminal blind spot of wrist cameras. ContactGuard rolls an action-conditioned latent world model forward through that blind spot to abort doomed grasps. We borrow its style of predicting in latent space from pooled multi-view embeddings. We also take its warning that joint-angle inputs create shortcuts in low-data regimes. ContactGuard only monitors and never explores, whereas we use a latent predictor to shape exploration. RVT and Perceiver-Actor avoid perspective occlusion through 3D re-projection, but both learn by imitation and include no exploration mechanism.
The sparse-reward literature supplies the competing ways to densify reward. LaNE rewards closeness to demonstration states in a DINOv2 feature space. VLAC is a pair-wise progress critic trained on about 40 million instances. It takes 30% to 90% success in 200 real episodes, but assumes progress rises over time and has no guarantee against background noise. SARM relies on hand-annotated stages. Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks gets random-network-distillation curiosity to work for door opening only by feeding it privileged simulator state. This is exactly the escape our method tries to replace with pixels.
Research Program
Let $z_{t+1}$ be the target: a latent embedding, from a slowly updated target encoder, of all cameras at step $t+1$, with patches covered by the robot's rendered silhouette removed. The silhouette comes from known kinematics and camera calibration. The policy model $f^{A}$ predicts $z_{t+1}$ from the policy's own camera stream and action $a_t$. The informed model $f^{\star}$ predicts the same target from all cameras, a short memory of past latents, the silhouette masks, and $a_t$. It is used only as a training-time critic in simulation or in a multi-camera lab. With per-step errors $e^{A}_t=\|f^{A}(\cdot)-z_{t+1}\|^2$ and $e^{\star}_t=\|f^{\star}(\cdot)-z_{t+1}\|^2$, the curiosity reward is the learning progress of the informed error:
$$r^{i}_t = \hat e^{\star}_{\text{prev}}(s_t,a_t) - e^{\star}_t,$$
where $\hat e^{\star}_{\text{prev}}$ is an LPM-style error model's estimate of what the previous checkpoint of $f^{\star}$ would have scored on this transition. The occlusion gap
$$g_t = e^{A}_t - e^{\star}_t$$
measures information the policy's camera lacked. Because the two models share a target, a positive gap cannot come from target noise; it can only come from missing inputs. The gap is never paid as curiosity. In an optional second stage, it becomes the reward for a camera-positioning action. This answers AV-ALOHA's open question of how to learn viewpoint control without demonstrations.
The program builds a shared off-policy image-based RL agent with the extrinsic +1 reward and adds four interchangeable reward modules: raw error, ICM-style inverse features, LPM on $f^{A}$, and our $r^{i}_t$. It instruments every run to log how much intrinsic reward falls within a few steps of occlusion onset or reveal. A reveal is defined from silhouette-target overlap, which is computable exactly in simulation. It also builds two controlled distractor families: a textured wall panel playing random video (a classic noisy TV), and an "action-coupled" variant that perturbs the target's pose only while it is hidden behind the gripper.
Baselines
The naive reference is the same agent with extrinsic sparse reward only, which shows how much any curiosity helps. The prediction-error baseline pays raw $e^{A}_t$. It is the standard curiosity form and should show the trap most plainly. The inverse-dynamics baseline trains features to predict the action, as in ICM. It is the classical defence against passive noise and the one our diagnosis predicts will fail on the self-occlusion case.
The strongest competitor on the noisy TV side is LPM computed on the policy's own camera. It is the only method in the file with demonstrated noise robustness. It reports that its exploration coverage dropped only 12.4% under injected MountainCar noise, against up to 85% for baselines, and that it reached Montezuma's Revenge rewards in 20 million steps versus about 50 million for RND (Beyond Noisy-TVs). No reported result was found on Lacuna for LPM, RND or ICM trained from pixels on RLBench or Robosuite, so we establish these numbers ourselves.
On the densification side, the strongest method is LaNE, run from its released code (PhilipZRH/LaNE) with its 1–5 demonstrations. Its paper reports outperforming E2C, MoDem and CoDER in learning speed and final success, but no raw percentage was found on Lacuna. LaNE uses demonstrations that our method does not, so beating it is not required. Matching it without demonstrations would be the strong result.
An upper bound is curiosity on privileged simulator state, following Bjelonic et al., who report 99% on push doors and package sorting. The ablation that isolates our key ingredient is raw error on $f^{\star}$ with no learning progress, alongside LPM on a multi-view $f^{A}$ that has no memory or masks. Together these separate "more information" from "progress gating."
All methods share the encoder, RL learner, replay size, environment-step budget, five seeds per task, and an equal hyperparameter search of twelve configurations for the intrinsic-reward coefficient and error-model rate. Beating LPM means higher success at a fixed step budget, or reaching LPM's final success in substantially fewer steps, with non-overlapping seed intervals, on occlusion-heavy tasks and on the distractor variants.
Why not just…?
Why not just mask out the robot's pixels before computing curiosity? Masking removes novelty from the arm's own appearance, but not the reveal surprise, which lives in the object pixels that reappear changed. We include masking inside every baseline's target, so any remaining gain has to come from the informed predictor. If masking plus LPM already closes the gap, the self-occlusion trap is smaller than claimed, and we would report that.
Why not just add cameras or use 3D re-projection like RVT? Adding cameras is a real threat. If one of several fixed views almost always sees the contact region, multi-view LPM without memory could match us. AV-ALOHA's static-camera results (30% needle threading, 8% pouring) and the terminal blind spot documented in VFAS-Grasp suggest that all views often fail at once near contact. That is why $f^{\star}$ includes memory. The ablation "multi-view LPM without memory" decides the question. RVT and Perceiver-Actor are imitation learners, and re-projection cannot fill a region no sensor saw.
Why not just use privileged state curiosity in simulation? It works, with 99% reported by Bjelonic et al., but it requires hand-chosen state for each task and does not transfer to pixel-only deployment. Our informed model also uses extra information at training time, but only more pixels and the robot's own geometry, which a multi-camera lab can supply. We treat privileged curiosity as the ceiling and report the fraction of it we recover.
Why not just densify with a progress critic like VLAC or demonstrations like LaNE? These change the problem by adding large pretraining corpora or demonstrations. VLAC also assumes progress rises over time, which retries break. We run LaNE and a VLAC-style pair-wise critic on the video-distractor variant. If they are unaffected and data are available, the honest conclusion is that curiosity matters mainly in the no-demonstration regime, and we would state that scope.
Proposed Experiments
The decisive first check is diagnostic. In Robosuite (block lift, drawer, peg insertion) and a subset of RLBench tasks with sparse +1 rewards, we measure what fraction of raw-error and LPM intrinsic reward arrives within five steps of occlusion onset or reveal. We compare that fraction to the fraction of time spent in those windows. If the reward is concentrated in occlusion windows, and the concentration survives LPM, the premise holds. If it does not hold even on the action-coupled perturbation variant, the proposal is wrong about manipulation, and we would pivot to studying ambient distractors only.
The second experiment is the "occlusion farming" test on the action-coupled variant: how often does each agent park its gripper between camera and target? Our method should drive this toward the no-curiosity rate. The third is the main comparison on clean tasks, the wall-video variant, and the action-coupled variant: success versus environment steps, plus the reward-at-occlusion statistic. The key prediction is that LPM holds up on the wall video but degrades on the action-coupled variant, while our method holds up on both.
The fourth experiment tests whether the occlusion gap $g_t$ is informative. On AV-ALOHA-style occluded insertion and hidden-pick tasks in simulation, we reward a camera arm with $g_t$ and compare against a fixed camera and a randomly moving camera. AV-ALOHA's demonstration-driven 30%→52% on needle threading gives a reference for what active vision can buy. Finally, we check that $r^{i}_t$ goes to zero on the wall-video-only environment, the empirical counterpart of the boundedness condition from BAMDP Shaping.
References
- Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring
- BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- VFAS-Grasp: Closed Loop Grasping with Visual Feedback and Adaptive Sampling
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- RVT: Robotic View Transformer for 3D Object Manipulation
- Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
- Accelerating Visual Sparse-Reward Learning with Latent Nearest-Demonstration-Guided Explorations
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
- Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks