Spatio-Temporal Token Budgets for Video Language Models

This direction is split into other directions: spatial-temporal resolution trade-off (spatio-temporal resolution trade-off). Its last version stays readable and citable here.

Version 1 of 1318 papers375 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are fundamentally limited by their context window—the maximum number of "tokens" they can process at once. When we move from text to video, this limitation becomes a crisis. A single high-resolution image can easily consume hundreds of tokens; a 10-minute video at standard frame rates would require millions.

This creates a "zero-sum game" for researchers: every token spent on higher spatial resolution (seeing a small object clearly) is a token that cannot be spent on temporal resolution (seeing how that object moves over time). The research direction emerging across these papers is the systematic management of this spatio-temporal trade-off.

The Performance Cliff and the Zero-Sum Game

The core problem is best illustrated by the "performance cliff" identified in MileBench. While models excel at single-image tasks, their accuracy drops sharply as the number of images increases. This isn't just a memory issue; it's a representation issue. As Yifan Du and colleagues argue in Exploring the Design Space of Visual Context Representation in Video MLLMs, visual context is a constrained optimization problem.

If you have a fixed budget of, say, 2,048 tokens, you can represent 4 frames with 512 tokens each (high detail, poor motion) or 64 frames with 32 tokens each (smooth motion, blurry objects). Most models currently fail because they use a "one-size-fits-all" sampling rate that is either too sparse to catch a quick gesture or too low-resolution to read a line of text.

Dual-Pathway Architectures: Mimicking Human Vision

One of the most successful strategies to break this trade-off is the dual-pathway approach, inspired by the human visual system's "what" and "where" pathways.

In SlowFast-LLaVA, Mingfei Gao and Mingze Xu propose a training-free baseline that uses a "Slow" path to capture high-resolution spatial details at a low frame rate and a "Fast" path to capture high-frequency motion at a lower resolution. Similarly, VideoGPT+ by Hanoona Abdul Rasheed integrates a dedicated image encoder for fine-grained detail with a video encoder for global context. By decoupling these two dimensions, models can "see" the small object and "understand" its trajectory without the tokens of one drowning out the other.

Dynamic and Adaptive Sampling

Rather than fixing the spatio-temporal ratio at training time, newer models like Dynamic-VLM by Haiyang Yu adjust the number of tokens per frame based on the video's length. This allows the model to prioritize spatial detail for short clips and temporal breadth for long ones.

A more surgical approach is found in SlowFocus. Ming Nie and his team developed a "zoom-in" mechanism: the model first scans the entire video using sparse sampling to find relevant segments, then re-samples those specific moments at a much higher density. This mimics how a human might skim a security tape and then pause to look closely at a specific face.

Moving Beyond the Frame

A subtle but profound shift in this field is the move away from the "frame" as the fundamental unit of video. Traditional models sample full frames, but Seong Hyeon Park argues in K-centered Patch Sampling for Efficient Video Recognition that this is a legacy of convolutional neural networks.

By sampling individual patches across the entire video duration—essentially picking the most informative "puzzle pieces" from different times and locations—transformers can achieve higher accuracy within the same token budget. This is pushed to the extreme in LLaMA-VID, which compresses an entire frame into just two tokens, and xGen-MM-Vid, which represents an entire video sequence using only 32 tokens.

Why This Works

The reason these strategies work is that video data is highly redundant. Most pixels in a video don't change from frame to frame, and most frames don't contribute to the core narrative of a scene. By using dual encoders, dynamic sampling, or patch-based selection, researchers are effectively "compressing" the video into the most semantically dense tokens possible.

The goal of this research direction is to move from "brute-force" video processing—where every pixel is treated as equally important—to an "intelligent" representation that allocates the LLM's precious context window to the specific details (spatial or temporal) that matter for the task at hand.

Papers in this version

Showing 60 of 318 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatio_temporal_token_budgets_for_video__e6da3330,
  title = {Spatio-Temporal Token Budgets for Video Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-token-budgets-for-video-language-models/rdr_e6da3330b0d84e86b81aa932aff42ac6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatio_temporal_token_budgets_for_video__e6da3330_v1,
  title = {Spatio-Temporal Token Budgets for Video Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-token-budgets-for-video-language-models/rdr_e6da3330b0d84e86b81aa932aff42ac6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}