NullTok: empirical-null token selection for training-free long video understanding

Zhang Ziyue, Juwei Jian, Baiyang Song, Yunshan Zhong, Jun Peng, Yiyi Zhou, Rongrong JiarXiv 2026 · arXiv

TL;DR

NullTok is a training-free approach for long video understanding that solves the "needle-in-a-haystack" problem of finding relevant visual information. By calibrating visual token importance against a "null distribution" of irrelevant queries, it identifies query-specific evidence across thousands of frames, allowing Video LLMs to process long videos with high accuracy while staying within strict token budgets.

What Problem Does This Paper Address?

Multi-modal Large Language Models (MLLMs) are powerful, but they have a "context window" problem: they can only ingest a limited number of visual tokens. For a long video, you can't just feed in every frame.

Current solutions usually fall into two camps:

  1. Frame Selection: Pick a few "important" frames and keep all their tokens.
  2. Token Pruning: Pick frames first, then discard less relevant patches within those frames.

The authors argue that both are flawed. Frame-level decisions are too coarse; a frame might have a low overall relevance score but contain a tiny, crucial detail (like a specific object or a brief action) needed to answer a question. If that frame is discarded early, that evidence is lost forever. Conversely, raw similarity scores between a query and a video patch are often "noisy"—a visually busy patch might score highly against any query, leading the model to select redundant or irrelevant data.

Methods

NullTok introduces a global, joint spatiotemporal selection strategy. Instead of picking frames first, it assesses every single token across the sampled video before deciding what to keep.

The core innovation is Empirical-Null Calibration:

  • The "Null" Reference: For every visual patch, the system calculates similarity scores not just for the target question, but for 64 "control" queries (irrelevant phrases from other videos).
  • P-Value Filtering: It treats these 64 scores as a baseline (the "null distribution"). If the target query's score is higher than almost all the control scores, it's likely query-specific evidence. This is converted into a Monte Carlo p-value.
  • Semantic Guard & Deduplication: To ensure nothing is missed, it combines this p-value filter with a "semantic guard" (keeping the top 10% of raw scores). It then uses a breadth-first search (BFS) algorithm to remove visually redundant tokens that are too similar to their neighbors.

Finally, a Joint Spatiotemporal Allocator distributes the token budget. Unlike rigid frame-selection methods, NullTok can choose to keep just a few tokens from many different frames or many tokens from a few key frames, depending on where the evidence lies.

Figure 1. Method schematic of NULLTOK. Target and control queries are evaluated on identical video coordinates. Calibration precedes deterministic redundancy reduction and joint spatiotemporal selection.

Datasets

The authors evaluated NullTok on three major long-video benchmarks:

  • Video-MME: A comprehensive benchmark covering short, medium, and long videos across six domains.
  • LongVideoBench: Focuses on referring reasoning with interleaved video and subtitles.
  • MLVU: A multi-task benchmark for long video understanding.

They tested the method using two "frozen" (unmodified) backbones: LLaVA-Video-7B and Qwen3-VL-8B.

Key Insights and Ideas

  • Calibration is King: Raw affinity scores are deceptive. By comparing a query's score to a "null" baseline at the exact same spatial position, NullTok filters out "loud" visual noise that isn't actually relevant to the question.
  • Avoid Early Commitment: By pooling all tokens from all frames into a single candidate pool before selection, the model avoids the "irreversible evidence loss" that happens when whole frames are discarded.
  • Sparsity is a Feature, Not a Bug: NullTok often selects very sparse tokens across a vast number of frames. This provides a much broader temporal context than traditional methods that pick a few dense frames.
Figure 2. Query-dependent allocation for three questions from one preregistered long video (rows 2289–2291). (a) Deduplicated candidate- token counts per sampled frame. (b) Four fixed snapshots and their deduplicated candidate masks before final allocation; hatched tokens indicate persistent semantic protection. (c) Final selected-token counts per sampled frame. Each question retains exactly 12,544 tokens, spanning 617, 712, and 686 nonempty frames for Q1, Q2, and Q3, respectively. The token budget is equivalent to 64 full 14 × 14 token grids, 64 × 196 = 12,544.

Key Results

NullTok sets a new state-of-the-art for training-free selection on Video-MME:

  • Performance: It achieved 65.81% accuracy with LLaVA-Video-7B and 68.85% with Qwen3-VL-8B at a budget of 64 Full-Frame Equivalents (FFE).
  • Efficiency: It outperformed all evaluated baselines, including complex "retrieve-then-compress" pipelines, under the same token budget.
  • Ablation: Removing the empirical-null calibration caused accuracy on LongVideoBench to drop from 64.40% to 61.03%, proving that the statistical calibration is doing the heavy lifting.
Figure 3. Video-MME accuracy with LLaVA-Video-7B at 64 FFE. NULLTOK exceeds the combination of QCA and MMTok by 0.74%. The vertical axis starts at 60%.

Limitations and Future Work

  • Preprocessing Overhead: While the authors note that dense features and control responses can be cached and reused for multiple questions about the same video, the initial global assessment of every token still requires a front-end computation step.
  • Encoder Dependency: The method relies on the quality of the frozen semantic encoder (they used DINO.txt). If the encoder can't "see" a specific attribute, NullTok won't be able to select it.
  • Statistical Scope: The authors acknowledge that while they provide a finite-sample guarantee for individual token screening, they do not yet claim family-wise error rate control over the entire final selection mask. Future work could involve more rigorous statistical bounds for the unified mask.

Summary updated

Abstract

Long video understanding remains a daunting challenge for existing multimodal large language models (MLLMs), mainly due to the large number of visual tokens accumulated in long videos. Currently, frame selection and token selection are two main paradigms for addressing this issue, both aiming to select relevant visual information to form a compact input for MLLMs. A natural approach combines these two paradigms by selecting frames before pruning tokens, but noisy frame-level relevance estimates can cause irreversible evidence loss. In this paper, we study this problem from the perspective of assessing the spatiotemporal distribution of query-specific evidence and propose a novel and training-free approach termed NULLTOK. For every patch across all frames, we construct an empirical-null distribution from 64 group-matched irrelevant control queries, capturing the scores expected in the absence of target-specific evidence. Target affinities are calibrated against these same-position references into right-tail Monte Carlo p-values, which are discrete super-uniform under a conditional exchangeability assumption. Calibrated screening and a high-recall semantic guard produce candidate tokens for deterministic visual deduplication and joint spatiotemporal budget allocation. To validate NullTok, we apply it to two representative MLLMs and conduct extensive experiments on a set of video benchmarks. The experimental results demonstrate its effectiveness under a bounded visual token budget. For example, it achieves 65.81% and 68.85% on Video-MME using frozen LLaVA-Video-7B and Qwen3-VL-8B backbones, respectively, at a cap of 64 full-frame equivalents (FFE), without subtitles or audio. These results outperform all evaluated baselines on Video-MME under the same budget for both backbones. Code is available at https://github.com/potato-kitty/NullTok.
BibTeX
@misc{ziyue2026nulltok,
  author = {Zhang Ziyue and Juwei Jian and Baiyang Song and Yunshan Zhong and Jun Peng and Yiyi Zhou and Rongrong Ji},
  title = {{NullTok: empirical-null token selection for training-free long video understanding}},
  year = {2026},
  eprint = {2611.23881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23881},
}
Figure