Robot policies built on vision-language models often fail in new scenes, and nobody can say why. The usual explanation tool is a heatmap over the camera image showing which pixels pushed the robot toward the action it took. That tool has a blind spot. Many decisions depend on something not being there: a clear path to the target, no lid on the pot, no second bowl that looks like the goal. When the policy's behavior hinges on an absence, there is no pixel to highlight. We propose to measure this kind of dependence directly. For each candidate object or condition, named in plain language, we add it to the scene in a realistic way and record how much the robot's intended motion changes. Adding a thing is well defined even when "removing" an absent thing is not.
The two-sentence version is this. A robot policy can rely on things being absent, and saliency maps cannot show that reliance. We expose it by inserting language-named concepts into the scene and measuring the shift in the policy's action distribution, check those scores against true interventions in simulation, and then use them to find and remove absence-based shortcuts that generic data augmentation leaves in place.
Core Claim
Negative evidence attribution has shown that image classifiers depend heavily on absent concepts. What is Missing? Explaining Neurons Activated by Absent Concepts finds that 2,036 of 2,048 final-layer ResNet-50 channels are significantly driven by inhibitory signals. In that work, inserting a neuron's "least activating" patch at the image border flips ImageNet predictions 84% of the time, against 8% for random patches, and debiasing that accounts for both presence and absence reaches 0.81 accuracy on inverted ISIC skin-lesion data, against 0.77 for presence-only debiasing. All of that work is on 2D classification. Its method assumes that concepts line up with single neurons, and its authors flag polysemantic units, where one neuron mixes several concepts, and generative outputs as open problems. The earlier contrastive explanation method, Explanations based on the Missing, defines a pertinent negative as a minimal absent feature whose addition switches the decision. It finds these by optimization against an autoencoder on MNIST and tabular data, and it concedes that continuous signals have no unambiguous "null" value.
Our claim is that the insertion half of the pertinent-negative idea carries over to robot policies, provided two things supply what those papers lacked. The first is a vocabulary of candidate absent concepts, which we take from language. The second is a way to realize each concept on the natural-image manifold, which we get from text-guided image editing and, in simulation, from the physics engine itself.
We make three specific, testable predictions. First, insertion scores should rank absent concepts by their true effect on rollouts much better than gradient saliency or random insertion. Second, a measurable fraction of vision-language-action (VLA) policy failures out of distribution should be explained by absence dependence and missed by presence-only tools. Third, augmentation targeted at the attributed absences should remove absence shortcuts at a lower data budget than the generic viewpoint and object augmentation studied in Shortcut Learning in Generalist Robot Policies.
Each direction plays a different role. Negative evidence attribution supplies the question and the insertion-flip test. Semantic-to-physical grounding supplies the setting: policies whose outputs are motions, not labels. Cross-modal transfer supplies the concept vocabulary and the bridge from words to pixels.
Research Program
The object of study is a VLA policy: a vision-language model fine-tuned to emit robot actions. Our main subject is OpenVLA, a 7B model with a fused SigLIP and DINOv2 image encoder. It outputs discretized 7-degree-of-freedom actions as tokens over 256 bins, so it gives an explicit probability distribution over actions. As a second subject we use a diffusion-headed policy in the spirit of DiffusionVLA, where the action distribution must be estimated by sampling. We keep this policy only if an open checkpoint is available; otherwise we train a diffusion policy on LIBERO ourselves.
For an observation $o$, instruction $\ell$, and candidate concept $c$ placed at location $r$, we define the absence dependence as
$$ A(c, r) = D\big(\pi(\cdot \mid o, \ell),\ \pi(\cdot \mid E(o, c, r), \ell)\big), $$
where $\pi$ is the policy's action distribution, $E$ is an operator that inserts $c$ at $r$ realistically, and $D$ is a divergence between action distributions. $D$ is total variation over OpenVLA's action bins, or for the diffusion policy the mean displacement of sampled end-effector waypoints. A large $A$ for a concept absent from $o$ means the policy's current behavior depends on that concept staying absent. We also compute the same quantity with $c$ already present and $E$ replaced by removal, which gives ordinary inhibitory attribution, where a present object suppresses the action. Together the two quantities cover the tug-of-war that the classification literature describes.
The concept vocabulary is built by cross-modal transfer. We generate candidate concepts in three ways and report which source finds more high-dependence concepts. The first source is object and affordance names drawn from the benchmark's asset library and from the affordance classes of MIFAG. The second source is objects that appear in other tasks of the same training set, since shortcut learning in fragmented data suggests these are the likely culprits. The third source is free-form concepts proposed by a language model from the instruction. MIFAG itself grounds only positive affordances, meaning where an action can happen. Our vocabulary deliberately includes obstructions, look-alike distractors and closures such as lids and doors, which play the role of negative affordances.
Pixel-level editing with a diffusion model is expensive, so we add a screening stage. Before editing, we rank candidates by shifting SigLIP patch features at $r$ toward the text embedding of $c$, following the visual-semantic projection idea of DeViSE. Only the top candidates go on to realistic pixel insertion. Screening in feature space is an efficiency trick, not the claim. It touches only the SigLIP half of the encoder and may move features off the image manifold, so we ablate it explicitly.
The insertion location $r$ is chosen along the policy's own predicted end-effector path and around the target. The method returns a ranked list of concepts and locations, which reads as "the robot is doing this because no X is here." We also evaluate the whole-image insertion variant, with no location, to see how much localization matters.
The debiasing stage turns attributions into training data. For each high-dependence concept that the task specification marks as irrelevant, we insert it into training frames and add a consistency loss that keeps the policy's action distribution unchanged. For each relevant concept, such as an obstruction on the path, we keep the demonstrated behavior only where simulation can relabel it. This is the embodied analogue of presence-plus-absence debiasing in Hesse et al.
Baselines
The naive reference point is the untouched policy, together with random insertion: realistic objects inserted at random locations, matched in count and editing pipeline. Random insertion is what gives Hesse et al.'s 84% versus 8% gap its meaning. If our method's flip rate is not far above random insertion, the language vocabulary and path-based placement contribute nothing.
For attribution quality, the strongest existing competitors are presence-based gradient explanations applied to the policy's action tokens. The first is Integrated Gradients on the log-probability of the executed action. The second is the class-contrastive back-propagation of "Why Not Other Classes?", adapted so that the contrast is between the executed action and an alternative action, such as the one reaching the distractor. The contrastive variant is the stronger of the two, because it at least asks "why this and not that." We run both on the same frames with the same number of policy forward and backward passes as our method, and we score all methods against the same ground truth.
For absence attribution at the neuron level, we run Hesse et al.'s non-target attribution and least-activating-patch insertion on the policy's vision encoder. This is the closest method in the file, and it isolates whether our concept-level insertion adds anything over neuron-level absence search.
For debiasing, the strongest competitor is the viewpoint and object augmentation of Shortcut Learning in Generalist Robot Policies. That paper reports that viewpoint augmentation reduced shortcut learning by 75% and improved out-of-distribution success by 175% in a real fine-tuning experiment, and that adding bridge objects raised one model's out-of-distribution success from 0.2 to 0.75 on the AgileX PIPER arm. Its reported SIMPLER object-augmentation figures (shortcut score 1.0 to 0.8, out-of-distribution success "from 0.68 to 0.25", described as an improvement) read inconsistently in the file, so we will not rely on them. We give generic augmentation exactly the same number of added training frames and the same fine-tuning steps as targeted insertion, and we tune both with the same hyperparameter budget. Beating it means lower shortcut score and higher out-of-distribution success at equal frame budget, or equal performance at a substantially smaller budget.
For raw task performance on real data, the reference numbers are OpenVLA's reported 70.6% success on BridgeData V2 (against 50.6% for RT-2-X) and 85% on the Google Robot platform. The file reports no LIBERO success numbers for OpenVLA, so we establish our own on LIBERO before any intervention.
The ablations that isolate the key ingredients are the following. We compare language-derived concepts against arbitrary asset objects. We compare pixel-level editing against feature-space shifts alone. We compare path-localized placement against random placement. Finally, we replace insertion scores with removal-only scores to measure how much signal comes specifically from absences.
Why not just…?
Why not just intervene in the simulator directly? In LIBERO you can place an object and re-run the rollout, which gives exact interventional answers. This is a real threat, and we do not argue around it: in simulation, direct intervention is the ground truth we evaluate against. The method matters in two places where simulation cannot help. The first is real robot logs, such as BridgeData V2 and DROID frames, where the scene cannot be re-staged. The second is the search itself, since exhaustively rolling out every concept at every location is far more expensive than scoring action distributions on single frames. The decisive experiment is rank correlation between single-frame insertion scores and full-rollout success changes in LIBERO. If that correlation is low, the method has no reason to exist off simulation, and we would report it as a negative result.
Why not just use saliency such as Integrated Gradients or Grad-CAM on the vision encoder? Reliable Evaluation of Attribution Maps in CNNs notes that such maps come back blank or noisy when a decision is driven by absence. Explaining Black-Box Automated Electrocardiogram Classification argues that masking-based perturbation cannot touch a component that is not in the input. We test this rather than assume it. We split the LIBERO ground-truth cases into presence-driven and absence-driven dependencies and report each method's recall on each split. We expect saliency to be competitive on the first split and near chance on the second. If it is not, the premise of the proposal is wrong.
Why not just add generic augmentation, as the shortcut-learning study does? The same study reports that uncontrolled diversity can worsen fragmentation. What is Missing? shows that presence-only debiasing leaves absence shortcuts partly intact (0.77 against 0.81 on ISIC). Neither result comes from robot policies. The budget-matched comparison in our shortcut experiments settles this for robot policies. If generic augmentation matches targeted insertion at equal frame budget, the debiasing contribution collapses to the diagnostic contribution alone, and we will say so.
Why not just ask a vision-language model what is missing from the scene? A captioning or critic model can list absent objects, but it reasons about the scene, not about this policy's internals. ManipBench also reports that even top models land between 60% and 80% on low-level manipulation questions, with means of 39.6% on articulated objects. We include this as a vocabulary source and as a standalone baseline: the language model ranks candidate absences by plausibility, and we score that ranking against rollout ground truth. We expect it to propose good candidates but rank them poorly, which is exactly why we use it only to feed the insertion test.
Proposed Experiments
The first experiment validates attribution against ground truth in LIBERO, using the Spatial, Object, Goal and Long suites from LIBERO. We fine-tune OpenVLA on each suite's 50 demonstrations per task. Then, for a held-out set of start states, we place each of several hundred candidate objects at path-localized and random locations inside the simulator and measure the change in rollout success and end-effector trajectory. That produces a ground-truth dependence table. Every attribution method scores the same single frames, and we report Spearman rank correlation with the table, precision at the top ten concepts, and the rollout flip rate of each method's top-ranked insertion. The flip-rate protocol mirrors Hesse et al.'s 84% versus 8% comparison. The decisive check is whether insertion scores beat contrastive gradients on the absence-driven split, while the pixel-level variant beats the feature-shift-only variant.
The second experiment is a controlled shortcut. We build LIBERO variants in which a spurious absence predicts the goal during training. For example, "put the bowl in the basket" is demonstrated only when a plate is absent, and "on the plate" only when the plate is present, with the instruction held fixed across an ambiguous subset. At test time we break the correlation. We ask whether each attribution method names the planted concept in its top results, and whether targeted-insertion debiasing restores out-of-distribution success faster than budget-matched viewpoint and object augmentation. We repeat the experiment on a SIMPLER setup matching the shortcut-learning paper so that its augmentation baseline runs on its home ground.
The third experiment is a failure census on real logged data. On BridgeData V2 frames where OpenVLA's predicted actions diverge from the demonstration, we run the method with image-editing insertion and classify each divergence as explained by an inhibitory presence, explained by an absence, or unexplained. Two annotators independently judge a sample of explanations for physical plausibility. The file reports no prior result for this census on Lacuna, so the target is descriptive: what fraction of divergences are absence-explained, and how many of those saliency misses.
The last experiment is a robustness check on the editing operator. Inserted objects can themselves be out of distribution, so we calibrate $E$ by inserting concepts the task specification says are irrelevant, far from the path. We report the resulting "editing noise" divergence and subtract it as a floor. If that floor is comparable to the scores of top-ranked concepts, pixel-level insertion is too noisy for real data, and the method's claims would be limited to simulation.
References
- What is Missing? Explaining Neurons Activated by Absent Concepts
- Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
- Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
- OpenVLA: An Open-Source Vision-Language-Action Model
- DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression
- Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding
- DeViSE: A Deep Visual-Semantic Embedding Model
- "Why Not Other Classes?": Towards Class-Contrastive Back-Propagation Explanations
- Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach
- Explaining Black-Box Automated Electrocardiogram Classification to Cardiologists
- ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning