Visibility Transfer Matrices for Auditing Robot Demonstrations

A robot arm that picks up or inserts an object usually loses sight of it at the moment that matters most. As the gripper closes in, the arm's own links and jaws block the cameras' view of the target. Policies learned from human demonstrations then act partly blind. This proposal applies an accounting tool from land-change science, the land use transfer matrix, to this problem. The matrix is a table that counts how much area moved from each category at one date to each category at a later date, and it separates land that stayed put from land that swapped. We apply the same table to the visibility state of the target object over a manipulation episode, with states such as fully visible, partly hidden by the arm, fully hidden, and in contact without sight. We then read the demonstration dataset the way an epidemiologist reads a retrospective patient registry: as a record whose contents were shaped by who chose what, not as a neutral sample.

The central claim is that many occlusion failures come from a coverage gap that aggregate success rates hide. Human teleoperators steer around some visibility transitions, and curated datasets keep only successful episodes. As a result, the demonstration data contain few or no examples of transitions the learned policy then enters often, such as partly hidden to fully hidden while still misaligned. Comparing the transfer matrix of demonstrations with the transfer matrix of policy rollouts, cell by cell, locates these gaps. Collecting new demonstrations aimed at the empty cells should then buy more success per demonstration than collecting the same number of extra demonstrations at random. Both halves of this claim can be measured, and either could turn out false.

Why this intersection, and what is actually new

Work on occlusion in manipulation attacks the blind spot by changing what the policy perceives or how it is built. RVT rebuilds the scene from point clouds and renders virtual views that remove the arm, reaching 62.9% average success on RLBench. C3DM crops a high-resolution window around a predicted contact point during diffusion denoising. DECO routes touch signals through a separate adapter so that they take over when vision drops out. FACT, from Demystifying When and Why VLAs Fail in Contact-Rich Tasks, reshapes the diffusion noise schedule and gates force inputs. ContactGuard predicts future latent states to flag failing grasps before contact.

All five treat the training data as given. None asks whether the demonstrations cover the occlusion situations the policy will actually encounter. That question is where this proposal differs. It is an analysis-and-data-collection method that sits alongside any of these policies, not a new policy architecture.

The retrospective-study literature supplies the lens. The clearest example is the pooled respiratory-failure analysis in Oxygenation/non-invasive ventilation strategy and risk for intubation. There, an apparent harm of non-invasive ventilation vanished once propensity weighting accounted for clinicians giving it to sicker patients (weighted odds ratio 1.10, p=0.562). That result motivates the idea that who chose an action shapes the recorded data.

The same literature also constrains the design. The pancreatic surgery study in Laparoscopic versus open pancreaticoduodenectomy stresses that matching balances only recorded variables. For that reason we do not propose reweighting logged demonstrations as the fix. Reweighting cannot create transitions that were never recorded, which is the overlap (or "positivity") failure in epidemiological terms. Datasets of successes only, such as DECO-50 (8,000 successful trajectories, resource), resemble registries that enroll only survivors.

The land-change literature supplies the method. Pontius-style intensity analysis, as applied in Intensity and Stationarity Analysis of Land Use Change Based on CART Algorithm, compares each transition's intensity against a uniform-intensity benchmark. It flags transitions that are more active than chance would predict. The karst study in Spatiotemporal evolution of karst rocky desertification shows why gross flows matter. Of the two opposing flows it reports, 11,433 km² of land degraded while 27,011 km² recovered, so the net figure hides far more churn in each direction than it shows.

We should be clear about the mathematics. A transfer matrix of counts is the same object as the transition counts of a first-order Markov chain. What we borrow is the accounting practice: comparing gross swaps with persistence, and testing intensity against a uniform benchmark. We do not borrow new mathematics. The conflict-index work in A Conflict Identification Method of Urban, Agricultural and Ecological Spaces relies on a hand-set 0.33 threshold, and the file names this kind of threshold as an open weakness. We replace it with a permutation test.

Core Claim

Label every timestep of every episode with one of five visibility states. For a source s, which is either demonstrations or rollouts, let $M^{s}_{ij}$ be the number of timestep pairs where state $i$ is followed by state $j$. From these counts we define a coverage gap for each transition:

$$g_{ij} = \frac{M^{\text{roll}}_{ij} / N^{\text{roll}}}{(M^{\text{demo}}_{ij} + \alpha) / N^{\text{demo}}}$$

Here $N^{s}$ is the total number of counted transitions in source $s$, and $\alpha$ is a small smoothing count. A large $g_{ij}$ marks a transition the policy enters far more often than the demonstrations ever showed it. We predict two things. First, rollout failures will be concentrated in high-$g$ off-diagonal cells, meaning transitions (swaps), more than in long stays on the diagonal (persistence). Second, spending a fixed budget of new demonstrations on episodes that pass through high-$g$ transitions will raise success more than random extra demonstrations or demonstrations aimed at high-occupancy states.

Research Program

We first build a visibility labeler. In RLBench simulation, the renderer's segmentation masks give exact per-camera visibility of the target, and contact is read from the simulator. On real logs such as DECO-50, we label with an off-the-shelf segmentation model run on multi-view frames, combined with the tactile channel for the contact states. Real-data labels are noisy, and we will report agreement against a hand-labeled subset.

Second, we compute demonstration and rollout matrices for a set of occlusion-heavy RLBench tasks under two policies. One is RVT, whose virtual views should shrink the occluded states. The other is a wrist-camera diffusion policy, which should suffer most. This design tests whether the method adds anything once occlusion has been partly removed at the perception level.

Third, we run intensity analysis at the transition level. We test which failure-associated transitions exceed uniform intensity, using a permutation test in place of a fixed threshold.

Fourth, we close the loop with targeted data collection. In simulation, a scripted expert is placed at start states that tend to produce the flagged transitions, with the arm-camera geometry reset to reproduce them. We compare against the controls described in the baselines below.

Fifth, we run a descriptive audit of DECO-50 to see whether success-only teleoperation shows the predicted empty cells, especially fully hidden to visible recovery transitions.

Baselines

The naive reference point is a policy retrained with the same number of extra demonstrations drawn at random from the expert's usual start-state distribution. The data budget, training compute, and hyperparameter search are identical to the targeted condition. Beating it means a statistically reliable success gain, with confidence intervals over seeds and at least several hundred evaluation episodes per task.

The strongest existing policy is RVT. On RLBench it reports 62.9% average success against 49.4% for PerAct (RVT). We will rerun RVT with the authors' training recipe on our task subset, rather than quote the average, because the subset differs. The claim to test is not that we beat RVT. It is that targeted demonstrations improve RVT itself more than random ones do.

For predicting which rollouts will fail, the strongest competitor is a ContactGuard-style learned monitor. ContactGuard reports ROC AUC 0.992 on Cup and 0.946 on Box pick-and-place in its real-robot suite (ContactGuard). No reported result on RLBench was found on Lacuna. We will train a latent-rollout-plus-linear-probe monitor on the same labeled rollouts that our matrix features use.

The ablation that isolates the key ingredient is state-occupancy targeting. It aims new demonstrations at the states where failing rollouts spend the most time, which uses only the matrix's row totals and ignores the off-diagonal structure. If this does as well as transition targeting, the land-use idea of separating swaps from persistence adds nothing.

A second, simpler control is a single scalar feature, the fraction of the episode spent occluded, used both as a failure predictor and as a targeting rule.

Why not just…?

Why not just add cameras or use virtual views so occlusion disappears? Terminal blind spots persist as the gripper closes on the target, even with active or mobile cameras, according to Active Vision Might Be All You Need and VFAS-Grasp as summarized in the file. The file also notes that RVT struggles when the gripper hides thin objects at contact. This is still a genuine threat. If RVT's rollout matrices show almost no mass in the occluded states, our method has little to act on. Running RVT as one of the two base policies settles the question directly, and we will report that outcome either way.

Why not just train a failure monitor like ContactGuard? A monitor tells you that an episode will fail, and the file itself calls such monitors abort mechanisms that cannot propose remedies. It does not tell you which data to collect. We compare prediction quality head to head. Even if the learned monitor predicts better, which is plausible, the decisive test is the data-targeting experiment, where the monitor has no direct counterpart. We will, however, also try targeting based on the monitor's highest-risk start states, as an extra competitor.

Why not just reweight the existing demonstrations, propensity-style? Reweighting can only rebalance transitions that exist in the logs. The pancreatic surgery analysis (Laparoscopic versus open pancreaticoduodenectomy) makes the parallel point that matching cannot fix what was never recorded. Where $M^{\text{demo}}_{ij}=0$, no weight helps. We include a reweighting condition anyway. It should help only on cells with small but nonzero counts, and seeing that pattern would support the overlap interpretation.

Why not just use a Markov model or the occluded-time fraction? The transfer matrix is a table of Markov transition counts. The question is whether looking at individual cells beats simpler summaries of them. The occluded-time and occupancy-targeting controls exist to answer exactly this. If they match transition targeting, we will report that the land-use framing reduces to occupancy statistics.

Proposed Experiments

On RLBench occlusion-heavy tasks, we first measure whether failure is better explained by cells with a large coverage gap $g$ than by occluded-time fraction. We score this with AUROC for failure prediction using rollout-level matrix features, cross-validated across seeds.

The decisive check is the equal-budget data experiment. For each base policy, we compare random, occupancy-targeted, monitor-targeted, reweighted, and transition-targeted demonstrations, across several budget sizes. The proposal stands if transition targeting beats both random and occupancy targeting at small budgets, and if the gain shrinks as $g$ flattens. The flattening part is the dose-response signature we expect if coverage is the mechanism.

A secondary check tests the retrospective-bias reading. We compare demonstration matrices from a scripted expert, which ignores visibility, with matrices from human teleoperation of the same tasks. If humans systematically avoid certain transitions that scripts pass through, that is direct evidence that operators' choices shape the dataset.

Finally, we run the DECO-50 audit. Its test is whether real success-only teleoperation shows near-empty recovery cells, and whether FACT or DECO failure modes reported on real hardware plausibly fall in those cells.