Probe Where You Can See: Occlusion-Aware Safe System Identification for Manipulation

A robot that must slide a puck to a target or balance a rod with an off-center weight first has to learn hidden physical properties, such as friction and where the mass sits. The most sample-efficient way to do this is to choose exploratory pushes that are maximally informative about those properties. That information, however, only arrives if the robot can see how the object responds, and the robot's own arm routinely blocks the camera exactly when it touches the object. Current exploration planners assume the outcome of every probe is observed. They will therefore happily choose pushes whose results are hidden behind the gripper. Those pushes yield no information, and they are also dangerous, because a violation such as the object sliding off the table or the rod tipping over cannot be detected while it is hidden.

This note proposes to treat visibility as a single quantity that governs both how much a probe teaches and how safely it can be executed. The robot's geometry, joint angles and camera pose are all known, so the planner can predict, before acting, how much of the object will be visible after each candidate push. We fold this predicted visibility into the information objective, so that hidden outcomes count for little, and into the safety constraint, so that uncertainty about a hidden object grows until it is seen again. The planner then picks probes that are informative, observable, and certifiably safe under the robot's belief about where the object is.

Core Claim

Under self-occlusion, the value of an exploratory action and its risk are coupled through the same predictable variable: how visible the object's resulting state will be. A planner that models this coupling explicitly should match unoccluded active system identification in downstream task success while using a single fixed camera, and should keep exploration-phase constraint violations near zero. Planners that ignore visibility will waste a measurable fraction of their probes on unobservable outcomes and incur violations they cannot see. Ignoring visibility in either the information term or the safety term alone should each produce a distinct, diagnosable failure.

Where This Sits

Active exploration for identifying physical parameters is well developed. FLEX picks actions that maximize a D-optimal criterion, meaning the log-determinant of the parameter Fisher information (a measure of how sharply the data pin down the parameters). It linearizes a neural dynamics model and solves each greedy step in about 4 ms. Optimal Exploration for Model-Based RL in Nonlinear Systems weights that information by its relevance to downstream control. ASID brings Fisher-information exploration to real manipulation, identifying friction and center of mass in a single real episode. These papers supply our method, but all three assume the relevant state is observed after every action. ASID in addition places no bound on what its exploratory slides and tips may do to the object or workspace.

Safe exploration supplies the constraint machinery. ActSafe pairs an optimistic bonus on dynamics uncertainty with a pessimistic cost estimate. SafeDreamer plans under cost constraints inside a latent world model learned from pixels. Backward Value Functions turn trajectory-level budgets into per-state constraints so that safety holds during learning. None of these targets parameter identification in contact-rich manipulation, and none distinguishes "the state is safe" from "the state is unobserved." Breaking Safety Paradox with Feasible Dual Policy Iteration constrains our design. It shows that safer policies starve cost models of boundary data, and informative probes in manipulation often live near boundaries such as table edges and tipping angles. We do not claim to resolve that tension. We only claim that the robot should approach a boundary while it can watch.

Occlusion work motivates the problem but solves a different one. AV-ALOHA adds a 7-DoF camera arm. This raises bimanual success under occlusion, but the added degrees of freedom hurt simpler tasks. RVT and ActiveVLA re-render point clouds from virtual viewpoints. Re-rendering cannot recover surfaces that no sensor ray reached. ContactGuard forecasts post-contact outcomes with a latent world model to veto bad grasps under wrist-camera occlusion, but it can only abort, not choose a better probe. We borrow ContactGuard's premise that one can reason ahead about occluded contact. We differ in using known robot geometry, rather than a learned probe, to predict visibility. This choice also sidesteps ContactGuard's finding that feeding joint states into a small-data learned model created a shortcut that hurt prediction: in our planner, proprioception enters only through exact geometric rendering.

Research Program

We model the object state $s$ (pose and velocity) as depending on unknown physical parameters $\theta$ and on a pushing action $a$. The object is tracked from a single RGB-D camera. For any predicted object state and arm configuration $q$, we render the known robot mesh from the camera to compute a visibility score $v(s,q)\in[0,1]$. This score is the fraction of the object's surface that is unblocked. The pose estimator's noise covariance $R(v)$ is calibrated offline as a function of visibility and grows sharply as $v\to 0$.

Over a short horizon of $H$ steps, the planner chooses pushes that solve

$$\max_{a_{1:H}} \; \log\det\Big(\Lambda_t + \sum_{k=1}^{H} J_k^\top R(v_k)^{-1} J_k\Big) \quad \text{s.t.} \quad \Pr_{b_k}\big[c(s_k,q_k) > 0\big] \le \delta \;\; \text{for } k=1,\dots,H.$$

In this objective, $\Lambda_t$ is the Fisher information accumulated so far. $J_k$ is the Jacobian of the predicted observed object pose at step $k$ with respect to $\theta$, obtained by automatic differentiation through the dynamics model as in FLEX. $v_k$ is the forecast visibility at step $k$. The constraint is evaluated under $b_k$, the robot's belief over the object state at step $k$. The cost $c$ encodes three hazards: contact force above a limit, the object leaving the table, and rod tilt beyond a recoverable angle. The belief is propagated with the dynamics model, and observation updates are weighted by $R(v_k)$. While the object is hidden, its covariance therefore grows without correction, and the chance constraint automatically becomes more conservative. We also impose a hard cap on consecutive steps at low visibility. This keeps the time the robot spends acting blind bounded.

The multi-step horizon matters. A push often occludes the object mid-stroke and reveals it on retraction, and a purely one-step planner in the style of FLEX cannot see that payoff. We solve the problem with the cross-entropy method, a sampling-based optimizer, over a low-dimensional push parameterization (contact point, direction, speed, stroke length), warm-started from the previous solution. The latency target is under 50 ms per decision.

The dynamics model will be a differentiable simulator with $\theta$ as physical parameters, following the structure of ASID. Once exploration ends, a downstream controller is trained or tuned in simulation using the identified $\theta$, again as in ASID.

The analysis has two parts. First, we will measure the "wasted information fraction": the share of predicted Fisher information that a visibility-blind planner attributes to actions whose outcomes turn out to be occluded. This quantifies the gap directly. Second, we will check whether the forecast visibility $v$ from mesh rendering agrees with the measured visibility in real images, because the whole approach rests on that prediction being accurate.

Baselines

The naive reference point is random exploratory pushes within the same safe workspace box, followed by the same downstream controller. Alongside it we run domain randomization over $\theta$ with no identification at all. ASID reports domain randomization at 0% (0/9) on real rod balancing and 30% on shuffleboard, and these numbers set the floor we expect to reproduce.

The strongest existing method for identification is ASID's Fisher-information exploration. ASID reports 66% (6/9) real rod-balancing success and 70% on shuffleboard. We run it with our exact perception stack, the same single camera, the same differentiable simulator, and the same number of exploratory interactions and tuning trials. It will be run twice: once with state estimated from the occluded camera, and once as an oracle with ground-truth object state, which gives the upper bound for unoccluded identification. We found no reported result on Lacuna for ASID's exploration-phase constraint violations, so we measure these ourselves.

The strongest safety competitor is ActSafe-style exploration: optimistic epistemic bonus plus pessimistic cost. We implement it on a SafeDreamer latent world model trained from the same camera, with the same interaction budget and hyperparameter search size as our method. This is the most serious "learn everything end-to-end" alternative. Its recurrent belief can in principle absorb occlusion implicitly. We found no reported result on Lacuna for ActSafe or SafeDreamer on contact-rich manipulation, so the Safety-Gymnasium result that SafeDreamer reduces cost by 94.3% relative to other model-based methods is context only, not a target.

A decoupled baseline runs ASID followed by a safety filter that rejects any push whose point-estimate prediction violates $c$. This filter mimics the veto-only role of ContactGuard.

Two ablations isolate the key ingredient. One sets $R(v)$ constant, so the planner is visibility-blind in information but keeps the belief-based safety constraint. The other keeps the visibility-weighted information but evaluates safety at the point estimate, ignoring covariance growth under occlusion. A third ablation sets $H=1$ to test whether multi-step lookahead is needed.

Beating the field means three things. Our method should reach downstream success within noise of the ground-truth-state ASID oracle. It should clearly exceed single-camera ASID. And it should record fewer exploration-phase violations than both ASID variants and the filter baseline, at equal interaction count.

Why not just…?

Why not just add more cameras, or re-render the point cloud from virtual views as RVT does? This is a real threat. If a second static camera removes occlusion in these tasks, our method's advantage shrinks to the cost of that camera. Re-rendering alone does not solve it: RVT and ActiveVLA transform whatever points were captured, and points behind the gripper were never captured. ActiveVLA's own ablation shows that fixed virtual views lose local detail, with success dropping from 91.8% to 87.6% without active selection and zoom. We settle the question empirically. The multi-camera oracle is in the study, and we also report a two-static-camera condition. If two cameras close the gap on every task, we will state that our contribution is confined to single-view and wrist-only setups, which remain common on real arms. A dedicated camera arm as in AV-ALOHA is excluded by design, because its extra degrees of freedom degraded simpler tasks there and add collision hazards of their own.

Why not just run ASID and bolt on a safety filter? A filter that sees only a point estimate treats "hidden" as "fine." It will pass pushes that send the object into the arm's shadow near a table edge. It will also reject informative pushes without offering the planner an informative alternative. The filter baseline and the point-estimate-safety ablation test exactly this. We expect the filter to show more violations that occur during occluded intervals, and lower information per interaction than our joint planner.

Why not just learn a latent world model from pixels and let curiosity plus pessimistic costs handle occlusion? A disagreement-based bonus in the style of ActSafe measures uncertainty in the model's predictions. Under occlusion, prediction uncertainty is high precisely because the outcome is unobservable, so the bonus may draw the robot toward hidden configurations rather than away from them. ASID already found that disagreement-based exploration (MAX) chased task-irrelevant dynamics, and FLEX reports 50 versus 500 samples against curiosity on cartpole. We will directly measure the fraction of ActSafe's exploratory steps spent at low visibility. If the latent approach nonetheless matches us on identification error and violations, that would be evidence that explicit visibility modeling is unnecessary, and we would report it as such.

Why not just use domain randomization and skip identification? ASID's 0% versus 66% on rod balancing already shows randomization fails when the task needs the true mass distribution. We rerun randomization only as a floor.

Proposed Experiments

The simulation study uses ManiSkill2, chosen for its fast rendering; it reports 2532 FPS with 64 environments. We build three tasks there: a rod-balancing task with an unknown center-of-mass offset, a shuffleboard task with unknown friction, and a push-near-edge task in which the most informative pushes send a puck toward the table boundary. Each task runs under three camera setups: one fixed shoulder camera, wrist camera only (which produces the terminal blind spot), and the multi-camera oracle.

The metrics are parameter error versus number of exploratory interactions; downstream task success; exploration-phase violations, counted and also split by whether they occurred while the object was hidden; the wasted-information fraction; and planning latency.

The decisive check has two predicted outcomes. With a single camera, visibility-blind ASID should show a large wasted-information fraction and a clear drop in success relative to its ground-truth-state oracle, while our method recovers most of that gap with near-zero violations. The two ablations should then fail in different ways. The constant-$R$ ablation should lose sample efficiency but stay mostly safe. The point-estimate-safety ablation should stay efficient but concentrate its violations inside occluded intervals. If neither ablation shows its predicted failure, the coupling claim is wrong. We will also plot success against camera count to locate where occlusion-aware planning stops mattering.

The real-robot study replicates ASID's rod-balancing protocol on a Franka Panda with a single RGB-D camera, using at least 20 trials per method rather than 9 so that differences are interpretable. We compare our method, single-camera ASID, and the filter baseline. The number to meet is ASID's reported 66% success with full sensing, and we additionally report every tip-over or table-edge event during exploration.