TL;DR
POKE (Probe Offline, Keep Efficiently) is a novel visual token pruning method for Vision-Language-Action (VLA) models that identifies which image patches are most critical for a robot's actions. By distilling "action sensitivity" from a full-graph teacher into a tiny student model, POKE achieves up to a $1.85\times$ inference speedup while maintaining significantly higher success rates than existing attention-based pruners.
What Problem Does This Paper Address?
Vision-Language-Action (VLA) models are the "brains" of modern robots, converting camera images and text instructions into physical movements. However, these models are computationally expensive because they process hundreds of visual "tokens" (small patches of an image) through deep Transformer layers.
To speed things up, researchers use token pruning—discarding less important patches early in the model. The challenge is deciding which tokens to keep. Most current methods use attention scores as a proxy for importance. But as the authors point out, what a model "looks at" (attention) isn't always what drives its "decisions" (action).
Furthermore, trying to calculate importance "on the fly" during inference leads to a phenomenon the authors call scoring-graph lock-in. If a token is pruned in one step, it loses its path to the action output in the computation graph. Consequently, its score drops even further in the next step, making it nearly impossible for the model to "re-discover" that part of the scene if it becomes relevant again.
Methods
The authors propose a two-stage approach to break the lock-in cycle:
- Offline Teacher (Action Sensitivity): Instead of attention, they define importance as action sensitivity: how much the final predicted action changes if a specific token is replaced by a neutral "mean" token. They estimate this efficiently using a single backward pass on the full, unpruned model using demonstration data.
- Lightweight Student Distillation: They train a tiny MLP student (less than 0.3M parameters) to predict these sensitivity scores directly from raw input features. Because the student sees the full image and doesn't rely on the VLA's internal state, it is immune to the "lock-in" effect.
- Inference with MMDP: At runtime, the student scores all tokens in a single fast forward pass. To avoid keeping redundant tokens (e.g., multiple patches of the same object), they use Max-Min Diversity Pruning (MMDP) to select a diverse set of high-scoring candidates.
Datasets
The researchers evaluated POKE across several challenging robotic manipulation benchmarks:
- LIBERO: Four suites (Spatial, Object, Goal, Long) covering various tabletop tasks.
- LIBERO-Plus: An extension of LIBERO designed to test robustness against shifts in camera viewpoint, lighting, and background.
- SimplerEnv: A simulated environment using the WidowX robot, where the student was trained on real-world BridgeData V2 frames and tested in simulation (real-to-sim transfer).
Key Insights and Ideas
- The "Lock-in" Discovery: The most significant "aha" moment is the empirical proof of scoring-graph lock-in. The authors showed that when scoring on a pruned graph, the model re-selects the same tokens 59% of the time (vs. 6% by chance), effectively becoming "blind" to the rest of the world.
- Action-Grounded Importance: Attention-based masks often focus on the robot's gripper or the main object but might ignore the "target" (like a plate the object needs to be placed on). Action sensitivity correctly identifies these distal but critical task elements.
- Efficiency through Distillation: By moving the "heavy lifting" of sensitivity analysis offline, the online overhead is negligible. The student model is so small that its inference time is nearly zero compared to the massive VLA backbone.
Key Results
POKE sets a new state-of-the-art for efficient VLA inference:
- Performance: On LIBERO, keeping only 32 out of 512 tokens, POKE achieved a 95.2% success rate on OpenVLA-OFT and 90.8% on $\pi_{0.5}$.
- Speedup: This corresponds to a $1.85\times$ and $1.26\times$ speedup, respectively.
- Superiority: It outperformed the previous best method, VLA-Pruner, by 5.9 and 15.75 percentage points at similar speeds.
- Robustness: In LIBERO-Plus, POKE maintained much higher success rates under visual perturbations compared to baselines, proving that action sensitivity is a more robust signal than attention.
Limitations and Future Work
- Offline Requirement: POKE requires a set of demonstrations to train the student model. While these are usually available for fine-tuned VLAs, it makes the method less "plug-and-play" for entirely zero-shot models.
- First-Order Approximation: Action sensitivity is calculated as a first-order derivative. This might miss complex, non-linear interactions between multiple tokens (e.g., the model needing two specific patches together to understand a spatial relationship).
- Future Directions: The authors suggest exploring real-robot evaluations and combining visual token pruning with other acceleration techniques like action-expert distillation.
Summary updated