POKE: Probe Offline, Keep Efficiently for Vision-Language-Action Models

Jaehyuk Lee, Yeajin Lee, Suhyun Bae, Donghun LeearXiv 2026 · arXiv

TL;DR

POKE (Probe Offline, Keep Efficiently) is a novel visual token pruning method for Vision-Language-Action (VLA) models that identifies which image patches are most critical for a robot's actions. By distilling "action sensitivity" from a full-graph teacher into a tiny student model, POKE achieves up to a $1.85\times$ inference speedup while maintaining significantly higher success rates than existing attention-based pruners.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models are the "brains" of modern robots, converting camera images and text instructions into physical movements. However, these models are computationally expensive because they process hundreds of visual "tokens" (small patches of an image) through deep Transformer layers.

To speed things up, researchers use token pruning—discarding less important patches early in the model. The challenge is deciding which tokens to keep. Most current methods use attention scores as a proxy for importance. But as the authors point out, what a model "looks at" (attention) isn't always what drives its "decisions" (action).

Furthermore, trying to calculate importance "on the fly" during inference leads to a phenomenon the authors call scoring-graph lock-in. If a token is pruned in one step, it loses its path to the action output in the computation graph. Consequently, its score drops even further in the next step, making it nearly impossible for the model to "re-discover" that part of the scene if it becomes relevant again.

Figure 1. Motivation on OpenVLA-OFT. (a) LIBERO-Goal, 64/512 tokens; rows: third-person/wrist; clear patches: retained. Attention misses the target plate kept by action sensitivity. (b) LIBERO-Long, B = 32, 9 closed-loop episodes (mean, IQR). Left: cumulative number of distinct tokens ever retained; pruned-graph scoring keeps returning to the same tokens. Right: share of score assigned to tokens outside the previous mask; 19% on the pruned graph vs. 77% on the full graph (dotted: uniform scores).

Methods

The authors propose a two-stage approach to break the lock-in cycle:

  1. Offline Teacher (Action Sensitivity): Instead of attention, they define importance as action sensitivity: how much the final predicted action changes if a specific token is replaced by a neutral "mean" token. They estimate this efficiently using a single backward pass on the full, unpruned model using demonstration data.
  2. Lightweight Student Distillation: They train a tiny MLP student (less than 0.3M parameters) to predict these sensitivity scores directly from raw input features. Because the student sees the full image and doesn't rely on the VLA's internal state, it is immune to the "lock-in" effect.
  3. Inference with MMDP: At runtime, the student scores all tokens in a single fast forward pass. To avoid keeping redundant tokens (e.g., multiple patches of the same object), they use Max-Min Diversity Pruning (MMDP) to select a diverse set of high-scoring candidates.
Figure 2. Method overview. (a) Offline training: a frozen full-graph VLA produces single-probe action-sensitivity labels, which are distilled into a lightweight student. (b) Inference: student predicts token scores, and MMDP selects diverse tokens from scoring candidates.

Datasets

The researchers evaluated POKE across several challenging robotic manipulation benchmarks:

  • LIBERO: Four suites (Spatial, Object, Goal, Long) covering various tabletop tasks.
  • LIBERO-Plus: An extension of LIBERO designed to test robustness against shifts in camera viewpoint, lighting, and background.
  • SimplerEnv: A simulated environment using the WidowX robot, where the student was trained on real-world BridgeData V2 frames and tested in simulation (real-to-sim transfer).

Key Insights and Ideas

  • The "Lock-in" Discovery: The most significant "aha" moment is the empirical proof of scoring-graph lock-in. The authors showed that when scoring on a pruned graph, the model re-selects the same tokens 59% of the time (vs. 6% by chance), effectively becoming "blind" to the rest of the world.
  • Action-Grounded Importance: Attention-based masks often focus on the robot's gripper or the main object but might ignore the "target" (like a plate the object needs to be placed on). Action sensitivity correctly identifies these distal but critical task elements.
  • Efficiency through Distillation: By moving the "heavy lifting" of sensitivity analysis offline, the online overhead is negligible. The student model is so small that its inference time is nearly zero compared to the massive VLA backbone.

Key Results

POKE sets a new state-of-the-art for efficient VLA inference:

  • Performance: On LIBERO, keeping only 32 out of 512 tokens, POKE achieved a 95.2% success rate on OpenVLA-OFT and 90.8% on $\pi_{0.5}$.
  • Speedup: This corresponds to a $1.85\times$ and $1.26\times$ speedup, respectively.
  • Superiority: It outperformed the previous best method, VLA-Pruner, by 5.9 and 15.75 percentage points at similar speeds.
  • Robustness: In LIBERO-Plus, POKE maintained much higher success rates under visual perturbations compared to baselines, proving that action sensitivity is a more robust signal than attention.
Figure 3. Qualitative rollouts on OpenVLA-OFT Long task 6 (B = 32) from the same initial state, showing frames and the applied third-person mask. VLA-Pruner times out at step 520 (red); POKE (top-k) succeeds at step 220 (green). Figure 4 adds more results.

Limitations and Future Work

  • Offline Requirement: POKE requires a set of demonstrations to train the student model. While these are usually available for fine-tuned VLAs, it makes the method less "plug-and-play" for entirely zero-shot models.
  • First-Order Approximation: Action sensitivity is calculated as a first-order derivative. This might miss complex, non-linear interactions between multiple tokens (e.g., the model needing two specific patches together to understand a spatial relationship).
  • Future Directions: The authors suggest exploring real-robot evaluations and combining visual token pruning with other acceleration techniques like action-expert distillation.

Summary updated

Abstract

Visual token pruning reduces the inference cost of Vision-Language-Action (VLA) models, but attention-based scores do not directly measure a token's effect on the predicted action. We define action sensitivity as the magnitude of the first-order action change when a token is replaced by its camera's mean token and estimate it for all tokens with a single backward pass. Computing it online is costly: scoring on the full (unpruned) graph needs an extra forward pass at every control step. Scoring on the pruned forward pass is cheaper but causes scoring-graph lock-in: removed tokens lose their direct paths through later layers and tend to receive lower scores, making those locations less likely to return. On replayed observations, even with a random conditioning mask, pruned-graph scoring re-selects 59% of the mask's tokens, versus 6% expected by chance; attention histories show a similar bias. POKE (Probe Offline, Keep Efficiently) computes full-graph action sensitivities offline and distills them into a student with under 0.3M parameters. At inference, the student scores the current observation without backpropagation or previous masks, and score-guided diversity selection retains the tokens. Keeping 32 of 512 visual tokens on LIBERO, POKE achieves 95.2% success on OpenVLA-OFT and 90.8% on $π_{0.5}$ at $1.85\times$ and $1.26\times$ speedups, outperforming attention-, diversity-, and distillation-based pruners, with larger margins on LIBERO-Plus.
BibTeX
@misc{lee2026poke,
  author = {Jaehyuk Lee and Yeajin Lee and Suhyun Bae and Donghun Lee},
  title = {{POKE: Probe Offline, Keep Efficiently for Vision-Language-Action Models}},
  year = {2026},
  eprint = {2611.21033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21033},
}
Figure