Instruction Guided Visual Token Compression

This direction is merged into another direction: semantic visual pruning. Its last version stays readable and citable here.

Version 1 of 116 papers23 conceptsfrozen 2026-10-04

Multimodal Large Language Models (MLLMs) face a fundamental "information bottleneck." While a high-resolution video or a 3D medical scan contains millions of pixels, a Large Language Model (LLM) has a finite context window and a high computational cost per token. Traditionally, researchers solved this by uniform downsampling—simply shrinking the image or skipping video frames. However, this "blind" compression often discards the very details needed to answer a specific question.

A new direction in efficient MLLMs replaces static downsampling with instruction-guided compression. The core intuition is that visual importance is not intrinsic to an image; it is dependent on the task. By using the user's prompt as a filter, these models can discard up to 98% of visual tokens while maintaining, or even improving, reasoning accuracy.

The Problem with Static Compression

When we process a video through a standard MLLM, we typically convert every frame into a grid of tokens. As Minbin Huang and colleagues note, the LLM is the most expensive part of the pipeline. Processing hundreds of redundant visual tokens leads to high latency and memory usage.

Static methods, such as fixed pooling or strided sampling, treat all pixels as equally important. If a user asks about a small license plate in a wide-angle shot, a static 2x2 pooling layer might blur that license plate into oblivion. To avoid this, models like Photon and HICom argue that the model should "know" what it is looking for before it decides what to forget.

Skimming the Visual Input

The mechanism behind this research is often described as "skimming." Just as a human might glance quickly at a crowd to find a friend in a red hat, instruction-guided models use the text prompt to identify "Regions of Interest" (RoI).

Yanan Liu and the team behind PAR (Prompt-Aware Token Reduction) demonstrate that treating visual tokens as a retrieval set allows for a massive reduction in computational load. By calculating the similarity between the user's question and the visual features, the model can prune away background tokens that do not contribute to the answer. Similarly, Chuanbin Liu and the HICom team use "instruction injection" to dynamically filter features, achieving state-of-the-art performance with nearly 80% fewer tokens.

Aggregation vs. Pruning

There are two primary ways to implement this guidance:

  1. Selective Pruning: Models like Photon (for 3D medical volumes) and PAR identify and discard irrelevant tokens entirely. This is highly effective for sparse tasks where the answer lies in a specific corner of the data.
  2. Dynamic Aggregation: Instead of discarding, models like PPLLaVA, developed by Ruyang Liu and Shangkun Sun, use the prompt to guide a pooling mechanism. They concentrate the "resolution" on important areas while summarizing the background into a few representative tokens. LEO-MINI, introduced by Yimu Wang, uses a Conditional Token Reduction (COTR) module to consolidate visual data into a fixed set of tokens based on text-aware similarity metrics.

Scaling Laws and the One-Token Limit

Perhaps the most surprising finding in this field comes from Sachin Goyal and J. Zico Kolter. Their work on "inference-optimal scaling laws" suggests that for many visual reasoning tasks, model size is more important than token count.

They developed QueCC, a compression algorithm that can condense an entire image into a single visual token. By conditioning this single token on the user's specific query, the model can still perform complex reasoning. This suggests that the "meaning" of an image relative to a question is actually quite low-dimensional, provided the compression is smart enough.

Why It Works: Semantic Alignment

The reason these methods succeed where static methods fail is semantic alignment. In models like LinVT, the text prompt acts as a query in a cross-attention mechanism. This ensures that the visual features passed to the LLM are already projected into the "thought space" of the question.

As Run Luo explains in the VCM (Vision Concept Modeling) framework, this mimics human selective attention. By transforming raw pixels into "concepts" relevant to the user's intent, MLLMs can finally handle long-form video and high-resolution imagery without the crippling computational tax of processing every redundant pixel.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_instruction_guided_visual_token_compress_e1126241,
  title = {Instruction Guided Visual Token Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instruction-guided-visual-token-compression/rdr_e11262412e3246a388e1c42abcd8eb7d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_instruction_guided_visual_token_compress_e1126241_v1,
  title = {Instruction Guided Visual Token Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instruction-guided-visual-token-compression/rdr_e11262412e3246a388e1c42abcd8eb7d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}