InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression

Haotian Ye, Qiyuan He, Jiaqi Han, Puheng Li, Jiaojiao Fan, Zekun Hao, Fitsum Reda, Yogesh Balaji, Huayu Chen, Sheng Liu, Angela Yao, James Zou, Stefano Ermon, Haoxiang Wang, Ming-Yu Liu

Abstract

Video tokenization is a critical component for bridging visual data with Large Language Models (LLMs). However, standard tokenizers employ a fixed compression rate, leading to significant redundancy in simple videos and information loss in complex ones. We present InfoTok, a principled framework for difficulty-aware adaptive video tokenization. Rooted in Shannon’s information theory, InfoTok utilizes the Evidence Lower Bound (ELBO) to estimate informational complexity and adaptively adjust token counts. Our method achieves up to 2.3x better compression than existing adaptive techniques and reduces token usage by approximately 50% compared to fixed-length tokenizers, while maintaining an 11x faster inference speed.

Introduction

Current vision foundation models typically represent video as a sequence of discrete tokens. A major limitation of existing tokenizers is their "one-size-fits-all" approach, where every video is compressed at the same fixed rate regardless of its content. This is inherently inefficient; for instance, a static scene requires far fewer tokens than a dynamic action sequence. While some flexible tokenizers have been proposed, they often rely on heuristic random masking or computationally expensive trial-and-error inference.

Methods

InfoTok addresses these inefficiencies through a framework built on three pillars:

  1. Information-Theoretic Grounding: Based on Shannon’s Source Coding Theorem, the authors demonstrate that optimal compression requires token length to be proportional to the data's negative log-likelihood (complexity).
  2. ELBO-based Router: To circumvent the intractability of calculating true log-likelihood, InfoTok employs the Evidence Lower Bound (ELBO) as a proxy. A router module estimates the ELBO to determine the target number of tokens ($N_x$) for a given video.
  3. Adaptive Compressor: A Transformer-based module prunes initial dense embeddings by discarding tokens with the lowest informational content (per-token ELBO), ensuring that critical visual features are preserved.
The InfoTok framework architecture, featuring an ELBO-based router and an adaptive compressor to prune embeddings based on informational complexity.

The authors also introduce InfoTok-Flex, a variant trained to handle multiple compression rates within a single model, allowing for on-the-fly bitrate adjustments.

Key Results

Evaluations on the TokenBench and DAVIS datasets demonstrate that InfoTok significantly outperforms both fixed-rate and prior adaptive tokenizers:

  • Efficiency: InfoTok saves ~50% of tokens compared to state-of-the-art fixed-length tokenizers like Cosmos-DV without sacrificing reconstruction quality.
  • Compression Performance: It achieves a 2.3x higher compression rate than ElasticTok at comparable quality levels, as measured by PSNR and BPP.
  • Inference Speed: By determining token length in a single pass rather than through binary search, InfoTok is 11x faster in terms of network evaluations (NFEs).
Rate-distortion performance comparison: InfoTok and InfoTok-Flex consistently outperform ElasticTok in PSNR across various bitrates (BPP).

Conclusion

InfoTok provides a theoretically grounded approach to adaptive video tokenization, transforming a heuristic engineering challenge into a principled optimization problem. By leveraging ELBO for complexity-aware pruning, it achieves superior efficiency and reconstruction fidelity. Future work includes integrating these adaptive sequences into large-scale generative models and extending the framework to other high-redundancy modalities like audio.

Summary updated

BibTeX
@inproceedings{ye2026infotok,
  author = {Haotian Ye and Qiyuan He and Jiaqi Han and Puheng Li and Jiaojiao Fan and Zekun Hao and Fitsum Reda and Yogesh Balaji and Huayu Chen and Sheng Liu and Angela Yao and James Zou and Stefano Ermon and Haoxiang Wang and Ming-Yu Liu},
  title = {{InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=JEYWpFGzvn},
}
Figure