Holding What the Arm Hides: Visibility-Aware Token Caching for Robot Policies

Robot policies that read camera images and output actions are expensive to run at control rate. The standard way to speed them up is to reuse computation for image patches that have not changed since the last frame. This reuse breaks exactly when precision matters most. As the gripper closes in on an object, the arm sweeps across the camera view and covers the target. The covered patches suddenly look different, so the cache throws them away and recomputes features of arm pixels. Those features say nothing about the object. The policy loses sight of its target at the moment of contact.

We propose a different rule for the cache, borrowed from the robotics idea of negative information: treating the absence of an expected observation as evidence. Because the robot knows its own joint angles, it can compute exactly which image patches its arm covers. Behind the arm, the cache should hold the last trustworthy features of the scene and let that trust decay slowly. Where the view is clear, or where the fingers have swept through space without touching anything, the cache should discount stale features that the new evidence contradicts. We encode this trust as a per-patch additive bias on the attention scores. Softmax normalization then automatically moves attention away from features that have been ruled out, the same way a belief distribution shifts probability mass away from regions confirmed empty. The hypothesis is that this makes caching both faster and more accurate than current caching under self-occlusion, without the latency of re-rendering virtual viewpoints.

Core Claim

Consider a vision-language-action model, meaning a transformer that takes camera patches plus a text instruction and outputs robot actions. Its visual key-value cache (the stored per-patch attention keys and values that can be reused across frames) should be governed by geometric visibility, not by how much a patch's appearance has changed. We maintain, for each cached patch $j$, a validity score $b_j \in [\epsilon, 1]$. It updates on three events:

  • Occlusion by the robot's own arm or gripper. The mask comes from forward kinematics projected into the calibrated camera. Here $b_j \leftarrow \lambda b_j$, a slow decay, and the patch's cached features are held rather than recomputed.
  • The patch is visible and matches its cache. Then $b_j \leftarrow 1$ and the patch is reused.
  • Negative evidence. Either the patch is visible and its content contradicts the held features, or the patch lies inside the projected volume the fingers swept through without registering contact. Then $b_j \leftarrow \epsilon$ and the patch is recomputed.

The policy's attention from query $i$ to cached patch $j$ uses the modified score

$$ s_{ij} = \frac{q_i^\top k_j}{\sqrt{d}} + \log b_j, $$

where $q_i$ and $k_j$ are query and key vectors, $d$ is the key dimension, and $\epsilon > 0$ is a floor. The floor handles detectors that miss objects: a wrongly discounted patch can recover weight when the object reappears, instead of being zeroed forever.

The pieces are simple. The combination is new. Current caches decide by appearance change, and current negative-information methods live in voxel grids. Here the same kinematic knowledge drives three things at once: skipping arm pixels, keeping occluded object features alive, and correcting those features when evidence says the object is no longer where the cache remembers it.

Where This Sits in Prior Work

The motivating tension comes from temporal caching. VLA-Cache reuses keys and values for patches whose cosine similarity to the previous frame is high. It evicts task-relevant patches using text-to-image attention. It reports that naively freezing static tokens drops success from 84.4% to 74.2% on LIBERO. That paper shows appearance-based caching needs a notion of task relevance. Its relevance signal is attention to the visible target, and that signal disappears when the gripper hides the target. We replace appearance-plus-attention with visibility-plus-negative-evidence.

DyCoke keeps pruned video tokens in a recycle bin for later restoration. It supplies the design pattern of retain-then-restore. However, it scores tokens by attention on 2D video question answering, with no geometry. We borrow the retention idea and make the retention decision geometric.

Negative information supplies the update logic. In Task-Driven Tactile Exploration, a hand that moves without touching anything rules out object poses intersecting its swept volume. That update runs over a discrete pose grid. BeliefMapNav discounts voxels that were in view but produced no detection, for room-scale navigation. Both work in explicit spatial grids, and neither touches transformer features. We move their updates into the cache of a learned policy.

The file's open-problems discussion of this literature also sets two constraints on the design. Distinguishing "occluded" from "confirmed empty" requires real geometry, which the robot's own kinematics provide cheaply for self-occlusion (though not for occlusion by clutter). And false negatives make hard zeroing pathological, which is why we use the floor $\epsilon$.

Occlusion-handling methods set the accuracy and latency references. ActiveVLA re-renders point clouds from chosen virtual viewpoints and zooms in. It reaches 91.8% average success on RLBench, at 0.53 s per step versus 0.26 s for a fixed view. Following RVT, it re-renders every step with no reuse across time. C3DM crops around predicted fixation points, again recomputed each step. ContactGuard imagines post-contact outcomes in a latent world model to veto bad actions. It only aborts; it does not maintain or correct state.

ContactGuard also reports that adding proprioception hurt, acting as a learning shortcut. That finding constrains our design. We use joint angles only outside the network, to compute masks and swept volumes, and never feed them as learned input. KeyWorld likewise assumes intermediate frames can be interpolated smoothly. Our displacement-under-occlusion tests target exactly the case where that assumption fails.

Research Program

We will build the mechanism on OpenVLA and OpenVLA-OFT, the backbones VLA-Cache used. That makes the comparison like-for-like.

Visibility mask. At each step we rasterize the robot's arm and gripper meshes into each camera using the current joint state and camera calibration. Patches more than half covered are marked occluded. Arm pixels are never recomputed. Their appearance is determined by the joint state, and their recomputation is what destroys ordinary caches.

Swept volume. Over each action chunk, we accumulate the union of the finger volumes. We project it to patches, and when the simulator or gripper reports no contact, we apply the negative update to held patches inside it.

"Contradicts the held features." This test is a cosine threshold on the patch embedding, applied only to visible patches. It is the one place appearance change still enters.

The analysis has two parts beyond success rate.

First, we will measure how well validity scores track ground truth. In simulation we know whether the held target features still describe the target's true location. We can therefore score $b_j$ as a predictor of staleness, reporting area under the ROC curve (the curve of true versus false detection rates across thresholds).

Second, we will sweep the decay rate $\lambda$ and the floor $\epsilon$ to map the trade-off between holding too long (acting on a moved object) and discarding too early (the VLA-Cache failure).

A real-robot replication on one arm with a fixed external camera will test sensitivity to calibration error. We will deliberately perturb the extrinsics by a few millimetres and degrees, because swept-volume reasoning is known to be fragile to exactly that (Task-Driven Tactile Exploration).

Baselines

Uncached backbone (naive reference). The first reference is OpenVLA (and OpenVLA-OFT) run on every frame with no reuse. It fixes the accuracy we must not lose and the latency we must beat. VLA-Cache reports 51.9 ms per step for this configuration on LIBERO.

VLA-Cache (strongest existing competitor). The strongest competitor on the efficiency-accuracy question is VLA-Cache. It is the only published cache designed for manipulation policies, and it already corrects the naive freezing failure. It reports 31.8 ms per step (1.63x faster), about 27% fewer FLOPs, and LIBERO success essentially unchanged (75.0% vs 74.7% as reported), with OpenVLA-OFT rising from 65 Hz to nearly 79 Hz.

We will rerun it from its released procedure on the same checkpoints, cameras, seeds and GPU. It gets the same hyperparameter search budget we give ours (similarity threshold and eviction ratio, matched against our $\lambda$, $\epsilon$ and contradiction threshold). We count as beating it matching its latency within 10% while exceeding its success on the occlusion-stress splits, with non-overlapping confidence intervals over seeds. We should also not trail it on standard LIBERO.

ActiveVLA (accuracy reference under occlusion). On RLBench, ActiveVLA is the accuracy reference for occlusion: 91.8% average success, 65.9% on COLOSSEUM, at 0.53 s per step. It is a different backbone and input representation, so we treat it as a reference point, not a matched baseline. The claim there is closing most of the gap between a fixed-view policy and ActiveVLA at fixed-view latency. We found no reported result on Lacuna for ActiveVLA, or any method, on an occlusion-only subset of RLBench tasks. We will compute that subset ourselves from released models where available.

Hold-only ablation. This ablation isolates the key ingredient. It uses the same kinematic mask and holds features behind the arm, with no decay, no swept-volume update and no attention bias.

No-swept-volume ablation. The full method with swept-volume updates removed. It separates visual from contact-based negative evidence.

Why not just…?

Why not just hold cached features behind the arm mask and skip the belief machinery? This is the real threat, because most LIBERO objects do not move while hidden. On such tasks, holding alone may capture nearly all the gain. We expect hold-only to match the full method on standard LIBERO. The decisive test is the displacement split described below, where the object is nudged or slips while covered. There, hold-only acts on stale features and should grasp at empty space.

If the two methods also tie on that split, the negative-information component is unnecessary. We would then report the simpler hold-only cache as the contribution.

Why not just give the policy a longer frame history and let it learn memory? Longer history multiplies visual tokens, working against the latency goal. It also relies on demonstrations containing examples of objects moving while occluded, which teleoperated LIBERO demonstrations rarely do. We will train a matched history-stacked OpenVLA-OFT with the same fine-tuning budget and compare success and latency on both splits.

Why not just add viewpoints: a wrist camera, a camera arm, or virtual re-rendering? Wrist cameras lose the contact region behind the fingers at close range, as VFAS-Grasp and Active Vision Might Be All You Need discuss. Dedicated camera arms add hardware. Virtual re-rendering doubles latency (ActiveVLA) and inherits depth artifacts on shiny or thin objects.

Our method is complementary rather than exclusive. LIBERO already provides a wrist camera, and all LIBERO runs use both views, so the comparison measures what visibility-aware caching adds on top of multiple views.

Why not just use a latent world model to imagine the occluded contact, as ContactGuard does? ContactGuard predicts failure well (ROC AUC 0.992 on its cup task). However, it outputs only an abort signal and adds a rollout per step. Our validity scores are a cheap, geometric version of the same "is my view of the target still trustworthy" signal, and they feed back into action selection. We will report both methods' staleness-detection accuracy on the displacement split as the head-to-head measure.

Proposed Experiments

Standard LIBERO. We evaluate all four LIBERO suites (LIBERO) under standard settings, reporting success over multiple seeds, per-step CUDA latency, FLOPs and control frequency. This establishes parity with VLA-Cache.

Occlusion-stress LIBERO. We move the external camera so the arm crosses the target on approach in most episodes, and fine-tune all methods on re-rendered demonstrations from that viewpoint.

Displacement-under-occlusion LIBERO. In the occlusion-stress setup, the simulator nudges the target by 2–5 cm at a random step while it is fully covered. This split is the decisive check for the negative-information component, run against hold-only and VLA-Cache. We will also log how often each method closes the gripper on empty space.

RLBench. We apply the mechanism to an RLBench-trained fixed-view policy on the 18-task set (RLBench). We report the gap to ActiveVLA's 91.8% and the latency ratio, and check robustness to distractors on COLOSSEUM.

Calibration stress. We perturb camera extrinsics and joint encoders to find the error level at which mask- and sweep-based updates start to hurt. We compare against the floor $\epsilon$ to show whether soft updates degrade gracefully.

Success criterion. The proposal succeeds if three things hold together: VLA-Cache-level latency, parity on standard LIBERO, and a clear gain over both VLA-Cache and hold-only on the displacement split.