PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

Shangkun Sun, Ruyang Liu, Haoran Tang, Yixiao Ge, Haibo Lu, Wei Gao, Jiankun Yang, Chen Li

TL;DR

PPLLaVA introduces a prompt-guided pooling strategy that slashes visual token counts by up to 18x without sacrificing performance. By aligning video features with specific user instructions, the model adaptively compresses redundant content, achieving state-of-the-art results in video understanding with significantly higher inference throughput.

What Problem Does This Paper Address?

Modern Video Large Language Models (Video LLMs) have become remarkably good at "watching" long videos, but they are computationally expensive. The standard approach is to feed every frame's visual tokens into the model, leading to massive context lengths that slow down inference and demand heavy hardware.

The authors argue that this is fundamentally inefficient because video content is highly redundant. For any given question, most of the video is "noise." For example, if you ask "How many times does the butterfly appear?", the model doesn't need high-resolution tokens for the parts of the video where there is no butterfly. Existing solutions like simple average pooling often lose critical temporal dynamics, while more complex "perceiver" modules (like Q-Formers) can be difficult to train and scale.

Figure 1: (a) An instance from VideoMME (Fu et al., 2024). The crucial information pertains to only a small portion of the video for different questions. (b) Performance comparison of PPLLaVA- LLaVA-Video with LLaVA-Video baseline on VideoMME-Overall (wo subtitles), which shows the number of visual tokens input to the LLM and the corresponding model performance.

Methods

The core of the paper is Prompt-guided Pooling LLaVA (PPLLaVA), which consists of three novel components:

  1. Fine-grained Vision-Prompt Alignment: The model uses a CLIP-based dual encoder to calculate an attention score between the user's text query and every spatiotemporal visual token. This creates a "relevance map" that identifies which parts of the video actually matter for the specific question.
  2. Prompt-guided Pooling: Instead of standard pooling, PPLLaVA uses the relevance map as a dynamic weight for a 3D convolution-style pooling operation. This allows the model to aggressively compress the video (up to 90% token reduction) while preserving the most important features.
  3. CLIP Context Extension: CLIP's text encoder typically has a very short context window (77 tokens), which is a bottleneck for complex, multi-turn dialogues. The authors implemented asymmetric positional embedding extensions, using different interpolation rates for different parts of the prompt to extend text capacity while preserving pre-trained knowledge.
Figure 2: The overview of PPLLaVA for compressing the video based on user prompts and generat- ing responses.

Datasets

The model was trained on a massive 1.16M sample multimodal dataset, including:

  • Images: LLaVA-1.5 (300k) and LLaVA-Interleave (594k multiple-image data).
  • Videos: Kinetics, SthSth-V2, Next-QA, CLEVRER, and LLaVA-Video-300k.

Evaluation was conducted across 7 major benchmarks, including Video-MME (for long-form video), NextQA and EgoSchema (for reasoning), and VideoChatGPT-Bench (for open-ended description).

Key Insights and Ideas

  • Redundancy is a Feature, Not a Bug: By identifying that most video data is irrelevant to a specific prompt, the authors turned a computational burden into a compression opportunity.
  • Instruction-Aware Compression: Unlike static pooling methods, PPLLaVA's compression is "smart." It changes what it "sees" based on what you ask. If you ask about a person's face, it preserves those tokens; if you ask about the background, it prioritizes those.
  • Plug-and-Play Versatility: The PPLLaVA architecture isn't tied to one specific model. The authors successfully integrated it into LLaVA-Next, LLaVA-Video, and InternVL3, showing consistent improvements across different base architectures.
  • Spatial vs. Temporal Sensitivity: The authors found that spatial pooling is much more "forgiving" than temporal pooling. You can compress spatial resolution significantly with minimal loss, but aggressive temporal compression (dropping too many frames) hurts performance more quickly.
Figure 3: Spatial pooling effects. We set T = 16 and kt = dt = 1, varying the spatial kernel size and stride.

Key Results

  • Efficiency: PPLLaVA achieves up to 18x token reduction. It maintains superior performance to the LLaVA-Video baseline while using only one-quarter of the tokens.
  • State-of-the-Art Performance: It set new records on several benchmarks. Notably, on Video-MME, it outperformed LLaVA-OneVision by 7.6% on videos longer than 30 minutes.
  • Generalization: Even on image-only tasks, the prompt-guided approach improved results, showing that "focusing" on relevant parts of an image is just as useful as it is for video.
  • Throughput: By reducing the token count, the model significantly improves inference speed, making it more viable for real-time applications.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Hardware Constraints: Due to limited GPU resources (8x A100s), they were unable to train models larger than 7B or 8B parameters. They suspect larger models would unlock even more potential from this architecture.
  • Prompt Dependency: Like many semantic-prior methods, the model's performance can dip if the user's prompt is extremely vague (e.g., just "Describe this video"). However, they noted that the model still performs well on captioning tasks by distributing attention across the foreground.
  • Future Directions: The team aims to explore training larger-scale models and further extending the context length to handle even longer video sequences (hours-long) more efficiently.

Summary updated

Abstract

In the past year, video-based large language models (Video LLMs) have achieved impressive progress, particularly in their ability to process long videos through extremely extended context lengths. However, this comes at the cost of significantly increased computational overhead due to the massive number of visual tokens, making efficiency a major bottleneck. In this paper, we identify the root of this inefficiency as the high redundancy in video content. To address this, we propose a novel pooling strategy that enables aggressive token compression while retaining instruction-relevant visual semantics. Our model, Prompt-guided Pooling LLaVA (PPLLaVA), introduces three key components: a CLIP-based visual-prompt alignment module that identifies regions of interest based on user instructions, a prompt-guided pooling mechanism that adaptively compresses the visual sequence using convolution-style pooling, and a clip context extension module tailored for processing long and complex prompts in visual dialogues. With up to 18x token reduction, PPLLaVA maintains strong performance across tasks, achieving state-of-the-art results on diverse video understanding benchmarks-ranging from image-to-video tasks such as captioning and QA to long-form video reasoning-while significantly improving inference throughput. Codes have been available at https://github.com/farewellthree/PPLLaVA.
BibTeX
@inproceedings{sun2026ppllava,
  author = {Shangkun Sun and Ruyang Liu and Haoran Tang and Yixiao Ge and Haibo Lu and Wei Gao and Jiankun Yang and Chen Li},
  title = {{PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=LOLhTA51tr},
}
Figure