Instruction-Guided Visual Token Pruning

This direction is merged into another direction: semantic visual pruning. Its last version stays readable and citable here.

Version 1 of 144 papers56 conceptsfrozen 2026-10-04

Multimodal Large Language Models (MLLMs) are notoriously computationally expensive, largely due to the "token explosion" that occurs when high-resolution images or videos are converted into sequences of visual patches. Traditionally, researchers tried to solve this by pruning "redundant" tokens based on visual saliency—discarding blurry backgrounds or repetitive textures.

However, a new wave of research suggests that visual importance is not a static property of an image. Instead, it is context-dependent. A patch of pixels representing a small ring on a table might be "redundant" for a general description of a room, but it becomes the most critical information if a user asks, "What color is the gemstone in the ring?" This shift from task-agnostic pruning to instruction-guided pruning allows models to discard up to 90% of visual data while maintaining near-original performance.

The Intuition: Text as a Filter

The core insight across this research is that the text prompt (the instruction or question) should act as a supervisor for the visual modality. In models like ELIP, Yangyang Guo and colleagues demonstrate that using language to guide visual token selection ensures that semantically relevant patches are preserved even if they appear visually insignificant in isolation.

This works by calculating cross-modal relevance. Instead of looking at how much a visual token attends to other visual tokens (self-attention), these methods look at how much the text tokens attend to the visual tokens. As Qingqing Cao and the authors of PuMer argue, the relevance of visual information is contingent on the specific textual query. If the text doesn't "look" at a patch, the model likely doesn't need it to generate an answer.

Mechanisms of Cross-Modal Selection

Researchers have converged on several distinct ways to implement this text-guided filtering:

1. Attention-Based Ranking

The most common approach uses the attention weights from the vision-language interface. In VISA, Pengfei Jiang and Fei Chao analyze attention weights from the text prompt to the visual tokens to prioritize patches the language model actively "queries." Similarly, TRIPS uses a text-based summary token to calculate attention scores, allowing Chaoya Jiang and Haiyang Xu to achieve a 40% speedup by pruning patches mid-way through the visual backbone.

2. Lightweight Policy Networks

Rather than relying on raw attention scores, which can sometimes be noisy, some frameworks use a "decider" module. AdaMTL, developed by Marina Neseem and Sherief Reda, uses a task-aware policy network to determine token relevance. In video settings, CaCoVID employs reinforcement learning to train a policy network that estimates which tokens actually contribute to a correct answer, rather than just which tokens are visually "loud."

3. Conditional Diversity and Coverage

Pruning too aggressively based on a query can sometimes lead to the loss of global context. To counter this, MMTok formulates selection as a "maximum coverage" problem. Sixun Dong and Qi Qian balance "text-vision coverage" (query relevance) with "vision-vision coverage" (global image representation). This ensures the model doesn't become so hyper-focused on the query that it loses the ability to understand the broader scene. CDPruner by Qizhe Zhang uses a similar logic, maximizing "conditional diversity" to retain task-specific details that might be visually subtle.

Why It Works: Breaking the Static Saliency Bias

The fundamental reason this direction is successful is that it addresses the static saliency bias. Traditional computer vision models are trained to find "objects," but human-AI interaction is often about "attributes" or "relationships."

As Hanning Chen notes in VLTP, task-oriented segmentation requires reasoning that static techniques lack. For instance, if an instruction asks to "find the chipped part of the plate," a standard pruner might discard the edges of the plate as "background." By integrating the instruction, models like ATP-LLaVA can adaptively adjust pruning ratios: a question about a tiny detail requires high token retention, while a general prompt allows for massive compression.

Efficiency Without Retraining

A significant advantage of many instruction-guided methods is that they are training-free and plug-and-play. SparseVLM and FlashVLM demonstrate that you can insert these selection mechanisms into existing pre-trained models like LLaVA. By using the text prompt to guide which parts of an image are important and "recycling" or merging the rest, Jusheng Zhang and Keze Wang show that models can actually outperform their unpruned counterparts by removing distracting, irrelevant visual noise.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_instruction_guided_visual_token_pruning_b5ef7e1b,
  title = {Instruction-Guided Visual Token Pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instruction-guided-visual-token-pruning/rdr_b5ef7e1b66e645efa5a393e87999a666},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_instruction_guided_visual_token_pruning_b5ef7e1b_v1,
  title = {Instruction-Guided Visual Token Pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instruction-guided-visual-token-pruning/rdr_b5ef7e1b66e645efa5a393e87999a666/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}