When a robot reaches for an object, its own arm and fingers often block the camera's view at the moment that matters most, just before contact. Today's systems handle this in one of three ways. Some always move a camera to look: an extra camera arm, or virtual re-rendering of a 3D scan. Some detect a likely failure and abort. Some act blind and hope. Each choice has a cost. Always looking is slow and makes learning harder. Aborting gives no recovery plan. Acting blind fails on precise tasks. This note proposes a different rule, borrowed from a classical idea in reinforcement learning called optimism in the face of uncertainty. The robot keeps a small set of learned predictive models, each a plausible account of what is hidden behind the occlusion. It executes the action that the most favorable still-plausible model says will succeed. If that action works, the robot has made progress. If it does not, the observed outcome rules that model out, and the next attempt is chosen from a smaller and more accurate set.
The twist that makes this safe for manipulation is an asymmetry. The robot is optimistic about task progress but pessimistic about irreversible damage. An action is allowed only if every plausible model agrees it will not knock over, drop, or crush something. Looking, meaning moving a camera or backing the arm off to clear the line of sight, is just another candidate action. It wins only when every progress-making action is vetoed by some plausible model. Looking therefore happens exactly when the robot cannot safely act through its uncertainty, and not otherwise. That gives a principled trigger for active vision, which the file identifies as missing. It also turns a pure failure detector into a recovery planner.
Core Claim
At test time, a chunked visuomotor policy proposes a handful of candidate action sequences. In that setting, the classically intractable optimistic planning step (maximizing value jointly over models and policies) becomes a cheap finite search over K models × N candidates. Two elements together should beat both "always look" and "veto and retry" on occlusion-heavy manipulation, without hurting easy tasks: within-episode elimination of models that contradict observations, and a worst-case damage veto. Concretely, let $C_t$ be the candidate chunks at decision step $t$, including a small library of look chunks. Let $S_t$ be the set of surviving world-model members. Let $p_k^{\text{succ}}$ and $p_k^{\text{dmg}}$ be member $k$'s predicted probabilities of task success and of irreversible failure after the chunk. The robot executes
$$a_t = \arg\max_{a \in C_t} \; \max_{k \in S_t} p_k^{\text{succ}}(z_t, a) \quad \text{subject to} \quad \max_{k \in S_t} p_k^{\text{dmg}}(z_t, a) \le \delta,$$
where $z_t$ is the encoded current observation and $\delta$ is a damage tolerance. After execution, it keeps only consistent members,
$$S_{t+1} = \{ k \in S_t : \lVert \hat z_{k,t+1} - z_{t+1} \rVert \le \tau \},$$
where $\hat z_{k,t+1}$ is member $k$'s predicted latent and $\tau$ is a threshold calibrated on held-out data. If $S_{t+1}$ empties, the system resets to the full ensemble, and we report how often this happens.
Relation to Prior Work
Three lines of work motivate the proposal by exposing the gap. ContactGuard rolls out a single action-conditioned latent world model (a predictor that forecasts future compressed observations given planned actions) ahead of contact and fires an abort when a probe predicts failure. It reaches ROC AUC 0.992 on its Cup task. It keeps one deterministic model and has no way to choose a replacement action. We reuse its architecture and latency profile, but replace one model with a consistency-filtered ensemble and replace the abort with a selection rule. CURA-PPO also predicts a distribution over outcomes under object-induced occlusion. However, it treats uncertainty purely as a cost inside model-free reinforcement learning, which produces risk-averse peeking. Our design is optimistic about progress and pessimistic only about damage, and it needs no reinforcement learning training loop. The hierarchical POMDP planner is the closest conceptual ancestor. It uses an optimistic (admissible) heuristic early to drive search under pose uncertainty and reaches 93% real-robot insertion success. But it assumes known deterministic dynamics and a hand-built belief over a static object, and it uses no vision. We learn both the hypotheses and the dynamics from demonstrations.
The optimism literature supplies the method and also constrains it. OMLE proves that optimistic selection over a confidence set is sample-efficient in partially observable problems only when observations are "weakly revealing", that is, when observations carry enough information about the hidden state. Total occlusion drives the relevant constant to zero and the guarantee collapses. Our response is a structural observation rather than a proof. In manipulation, how revealing the observations are depends on the action taken, so including look chunks lets the agent restore information on demand. We state this as a working hypothesis to test empirically. SOMBRL avoids the joint model–policy search by planning on mean dynamics with an uncertainty bonus. That is the right move when the policy space is continuous. In our setting the candidate set is finite, so the exact optimistic maximum is affordable, and we test directly whether it beats the bonus approximation. MEX and HOT-GP are alternative relaxations that operate across training episodes. Our elimination instead operates within an episode, over hypotheses about the currently hidden scene.
On the occlusion side, AV-ALOHA adds a third 7-DoF camera arm controlled jointly with manipulation. This helps on occlusion-focused tasks but hurts simple ones, because the 21-DoF action space adds variance. ActiveVLA selects virtual viewpoints on every inference, at 0.53 s per inference versus 0.26 s for a fixed view. Both look unconditionally. Our proposal decides when to look. DECO compensates for occlusion with touch. That approach is orthogonal: tactile or contact signals can enter our latent observations and sharpen elimination.
Research Program
The system is built in three parts. First, we train K latent world models, each a JEPA-style encoder plus an action-conditioned predictor in the manner of ContactGuard. The members differ in initialization, in bootstrap resampling of demonstrations, and in randomized masking of visible regions during training. The masking is meant to diversify how members complete occluded content. Each member carries two small probes, one for success and one for irreversible failure. The labels come from simulator ground truth: object dropped, knocked off the workspace, or collision force above a limit. Second, a frozen base policy proposes N chunks per step. This is an ACT policy, whose generative latent we sample to obtain diverse chunks. The candidates are augmented with a fixed library of look chunks: camera-arm repositioning in AV-ALOHA, or arm retraction to a clearing pose on a single-arm platform. Third, the selection and elimination rules above run at each chunk boundary.
The analysis component asks whether ensemble disagreement actually tracks occlusion. In RLBench we can compute the visible fraction of the target from segmentation masks. We measure the correlation between member disagreement on $p^{\text{succ}}$ and that visible fraction. If deep ensembles turn out to agree confidently about hidden content, the mechanism has nothing to work with. That would be an early and cheap failure signal, checked before any policy experiments.
Baselines
The naive reference point is the base ACT policy with static cameras, no monitor, and no looking. It shows what occlusion costs. AV-ALOHA reports 30% on Thread Needle and 8% on Pour Test Tube for static setups, and 40% for the static-only real Hidden Pick baseline.
The strongest existing method on the occlusion-focused tasks is AV-ALOHA's learned active-vision policy, which always controls the camera arm. Its reported results are 52% on Thread Needle, 14% on Pour Test Tube, and 60% on real Hidden Pick with AV plus wrist cameras (AV-ALOHA). We retrain it on the same demonstration set, with the same ACT backbone and the same hyperparameter search budget as our base policy. Beating it means exceeding 52% on simulated Thread Needle while also matching the static policy on Peg Insertion, Slot Insertion, and Hook Package, where AV-ALOHA itself degrades.
On RLBench, the strongest competitor is ActiveVLA at 91.8% average success across 18 tasks, falling to 89.4% without active viewpoint selection (ActiveVLA). Here our rule gates ActiveVLA's viewpoint stage instead of a physical camera arm. Success means matching 91.8% while invoking active selection on a minority of steps, which would recover latency toward the 0.26 s fixed-view cost.
The third baseline is veto-and-resample. A single ContactGuard-style model aborts a chunk predicted to fail, and the policy draws a fresh sample. It gets the same rollout budget as our method, K × N model calls per step. This baseline isolates whether a model set and elimination add anything beyond filtering.
The fourth baseline is SOMBRL-style selection, which ranks candidates by mean-ensemble success plus a scaled disagreement bonus. Its bonus weight is tuned on the same validation tasks with the same budget we use for $\delta$ and $\tau$.
The fifth baseline is greedy information gain: look whenever ensemble disagreement exceeds a threshold, otherwise follow the mean. This mirrors the greedy baseline that the POMDP planner beat by over 50% in solution quality.
Three ablations isolate the key ingredients. The first removes within-episode elimination, so the full ensemble is used at every step. The second removes the damage veto, giving pure optimism. The third keeps only pessimism, ranking by worst-case success; this approximates the risk-averse stance of CURA-PPO.
Why not just…?
Why not just abort and resample, as ContactGuard already enables? This is the most serious practical threat, because it is simple and fast. Its weakness is that a single deterministic model makes the same mistaken guess about the hidden scene on every retry. Resampled chunks are vetoed or accepted under one fixed belief. Elimination changes the belief after a contradiction. The decisive measurement is second-attempt success rate after a failed first attempt. If veto-and-resample matches us there, the ensemble is not earning its cost, and we would report the method as an ensemble-calibration study rather than a planner.
Why not just use an uncertainty bonus on the mean model, as SOMBRL does? SOMBRL's bonus exists to avoid an intractable joint search, and in a finite candidate set that search is not intractable. More importantly, a bonus rewards uncertainty wherever it appears. Under occlusion, uncertainty is high everywhere behind the hand, so a bonus favors uncertain actions whether or not they could lead to success. The optimistic maximum favors an action only if some plausible model predicts it succeeds. This is a genuine empirical question, and the fourth baseline answers it on the same budget. A tie would weaken the claim to "the veto and elimination matter, and the optimism form does not."
Why not just always look, with a camera arm or virtual views? The file documents the costs directly. AV-ALOHA's extra degrees of freedom hurt tasks that static cameras already solve, and ActiveVLA roughly doubles inference time. Looking also cannot see through closed fingers in the final inch. Our prediction is that gated looking matches always-looking on hard tasks and matches static on easy ones. The per-task look rate makes this visible.
Why not just add touch, as DECO does? Touch is the right signal once contact happens, and DECO's tactile adapter gives a 20% absolute gain on blind contact-rich tasks. But touch arrives only after commitment, and the damage we want to avoid often occurs at first contact. Our rule decides what to commit to. Where tactile or force signals exist, they enter $z_{t+1}$ and should speed elimination. This is complementary, not competing.
Proposed Experiments
The primary testbed is the AV-ALOHA MuJoCo suite. Its seven tasks split cleanly into standard tasks and occlusion-focused ones (Thread Needle, Occluded Insertion, Hidden Pick, Pour Test Tube). That split lets us test both halves of the claim: gains where occlusion bites, and no loss where it does not. The secondary testbed is RLBench's 18-task set, with ActiveVLA as the base policy. The camera-pose and distractor axes of THE COLOSSEUM then test whether elimination stays stable under viewpoint shift; ActiveVLA's reported 65.9% there is the reference. A small real-robot check on an AgileX Piper replicates the ContactGuard Cup and Box tasks. It asks whether converting aborts into selections raises task success without lowering the abort-quality AUC of 0.992 and 0.946.
Metrics are task success, irreversible-failure rate, look actions per episode, wall-clock time per episode, decision latency per step, second-attempt success conditioned on first-attempt failure, and ensemble-reset frequency. That last metric is a direct measure of how often the learned hypothesis set is misspecified.
Four checks would decide the question. The method should exceed AV-ALOHA's 52% on Thread Needle while staying within noise of the static policy on the three standard tasks. Its look rate should be near zero on standard tasks and concentrated just before contact on occlusion tasks. Its second-attempt success should clearly exceed veto-and-resample's under equal compute. And disagreement among members should correlate with measured occlusion of the target. If the last check fails, the rest of the program is moot, and we would say so.
References
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- Uncertainty-Aware Non-Prehensile Manipulation with Mobile Manipulators under Object-Induced Occlusion
- A POMDP-based hierarchical planning framework for manipulation under pose uncertainty
- When Is Partially Observable Reinforcement Learning Not Scary?
- SOMBRL: Scalable and Optimistic Model-Based RL
- Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration
- Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation