Abstract
Video tokenization is a critical component for bridging visual data with Large Language Models (LLMs). However, standard tokenizers employ a fixed compression rate, leading to significant redundancy in simple videos and information loss in complex ones. We present InfoTok, a principled framework for difficulty-aware adaptive video tokenization. Rooted in Shannon’s information theory, InfoTok utilizes the Evidence Lower Bound (ELBO) to estimate informational complexity and adaptively adjust token counts. Our method achieves up to 2.3x better compression than existing adaptive techniques and reduces token usage by approximately 50% compared to fixed-length tokenizers, while maintaining an 11x faster inference speed.
Introduction
Current vision foundation models typically represent video as a sequence of discrete tokens. A major limitation of existing tokenizers is their "one-size-fits-all" approach, where every video is compressed at the same fixed rate regardless of its content. This is inherently inefficient; for instance, a static scene requires far fewer tokens than a dynamic action sequence. While some flexible tokenizers have been proposed, they often rely on heuristic random masking or computationally expensive trial-and-error inference.
Methods
InfoTok addresses these inefficiencies through a framework built on three pillars:
- Information-Theoretic Grounding: Based on Shannon’s Source Coding Theorem, the authors demonstrate that optimal compression requires token length to be proportional to the data's negative log-likelihood (complexity).
- ELBO-based Router: To circumvent the intractability of calculating true log-likelihood, InfoTok employs the Evidence Lower Bound (ELBO) as a proxy. A router module estimates the ELBO to determine the target number of tokens ($N_x$) for a given video.
- Adaptive Compressor: A Transformer-based module prunes initial dense embeddings by discarding tokens with the lowest informational content (per-token ELBO), ensuring that critical visual features are preserved.
The authors also introduce InfoTok-Flex, a variant trained to handle multiple compression rates within a single model, allowing for on-the-fly bitrate adjustments.
Key Results
Evaluations on the TokenBench and DAVIS datasets demonstrate that InfoTok significantly outperforms both fixed-rate and prior adaptive tokenizers:
- Efficiency: InfoTok saves ~50% of tokens compared to state-of-the-art fixed-length tokenizers like Cosmos-DV without sacrificing reconstruction quality.
- Compression Performance: It achieves a 2.3x higher compression rate than ElasticTok at comparable quality levels, as measured by PSNR and BPP.
- Inference Speed: By determining token length in a single pass rather than through binary search, InfoTok is 11x faster in terms of network evaluations (NFEs).
Conclusion
InfoTok provides a theoretically grounded approach to adaptive video tokenization, transforming a heuristic engineering challenge into a principled optimization problem. By leveraging ELBO for complexity-aware pruning, it achieves superior efficiency and reconstruction fidelity. Future work includes integrating these adaptive sequences into large-scale generative models and extending the framework to other high-redundancy modalities like audio.
Summary updated