Natural language processing has flourished largely because text is inherently discrete. Words and characters form a finite vocabulary of distinct symbols, allowing models to easily compute probabilities over sequences, mask out tokens to learn context, and generate text step-by-step.
Video, by contrast, is a continuous, high-dimensional firehose of pixels. Operating directly on raw pixels is computationally prohibitive and forces models to waste valuable capacity on low-level visual noise—such as minor lighting shifts or compression artifacts—rather than high-level semantic events. As noted by the authors of Generative Video Transformer: Can Objects be the Words?, this lack of native discrete tokenization prevents standard Transformer architectures from tracking long-term structures or reasoning about scene dynamics efficiently.
To bridge this gap, a prominent research direction focuses on discrete video tokenization: transforming continuous visual data into a sequence of symbolic "visual words." By mapping pixels to a discrete latent space, researchers can treat video as a specialized language, unlocking the full power of modern sequence modeling.
From Clustering to Vector Quantization
The core challenge of video tokenization is defining a meaningful "vocabulary" for visual data. Early approaches tackled this by extracting hand-crafted features or local patches and grouping them using traditional clustering algorithms.
For instance, Bertini 0001 et al. quantized frame-level feature histograms into a visual vocabulary, allowing them to treat video clips as string-like structures for event classification. Similarly, Cheng et al. clustered local segments into discrete sub-events to capture long-range temporal dependencies, while Szlam et al. mapped continuous spatial patches to a finite set of centroids via k-means clustering to predict motion without the blurring typical of continuous pixel-space models.
Modern architectures replace these hand-crafted pipelines with deep generative models, such as Vector-Quantized Variational Autoencoders (VQ-VAEs) or VQ-GANs. Instead of arbitrary pixel groupings, these networks learn an end-to-end codebook of discrete latent vectors.
Continuous Video Frame ──> [ Encoder ] ──> Continuous Latent Vectors
│
(Quantization Step)
▼
Discrete Token Sequence <── [ Codebook Indices ] ◄─ Nearest Neighbor Search
During the quantization step, continuous feature vectors from spatial patches or frames are mapped to their nearest neighbor in the learned codebook. This discrete bottleneck forces the network to discard high-frequency pixel noise and retain only the essential structural and semantic information. As demonstrated by Sun et al. in their work on VideoBERT, this quantization shifts the paradigm from pixel-level analysis to discrete token sequences, making it possible to capture long-term temporal dependencies that low-level feature extractors often miss.
Unlocking the Language Modeling Toolbox
Once a video is represented as a sequence of discrete tokens, it can inherit the exact training objectives and architectures that revolutionized natural language processing.
Masked Token Prediction
In the style of BERT, models can be trained to predict missing parts of a video. Tan et al. utilize this in the VIMPAC framework, employing a block-wise masking strategy over a grid of visual tokens. By forcing the model to reconstruct masked visual blocks, it develops a deep, self-supervised understanding of spatio-temporal relationships.
Autoregressive Next-Token Prediction
Discretization allows generative video modeling to be framed as next-token prediction. Zhang et al. demonstrate in their work on VidIT that when large autoregressive Transformers are trained on discrete visual tokens, they develop emergent "zero-shot" video imitation capabilities. By simply prepending a demonstration video to a query clip, the model infers the underlying action and applies it to a new scene.
Sequence-to-Sequence Translation
Representing video as tokens enables direct cross-modal translation. In unsupervised sign language translation, Guo et al. use a VQ-VAE to convert continuous video frames into a sequence of codebook indices. This allows them to apply established NLP sequence-to-sequence techniques, such as back-translation, directly to video data, bridging the gap between high-dimensional gestures and discrete spoken words.
Cross-Modality Alignment and Semantic Anchoring
A major benefit of discrete visual tokenization is that it provides a shared mathematical language for multi-modal models. When visual data lives in a discrete latent space, aligning it with text embeddings becomes significantly more robust.
Ye et al. show that injecting discrete embeddings into a high-entropy visual feature stream acts as a "semantic anchor." This process reduces representational noise by pulling continuous visual features toward structured language tokens, boosting translation performance.
For multi-modal video generation, frameworks like MMVID (Han et al.) and MAGE (Luo et al.) leverage discrete token grids to seamlessly condition video generation on text prompts and static images, ensuring structural consistency across frames.
Furthermore, this alignment allows frozen, pre-trained language models to process video directly. Xu et al. introduce a learnable abstractor that summarizes lengthy video sequences into a fixed number of visual tokens, allowing a decoder-only language model to treat video features as a specialized dialect. Similarly, the authors of SceneLLM quantize encoded visual features into discrete tokens to construct a "scene sentence," enabling large language models to perform implicit spatio-temporal reasoning for dynamic scene graph generation.
Hierarchical and Hybrid Representations
While patch-based tokenization is highly effective, tokenizing every single spatial patch across time can still lead to long sequence lengths. To scale to longer videos, researchers have introduced hierarchical and hybrid tokenization strategies that abstract video at higher semantic levels.
One approach is to change what constitutes a token. Instead of uniform pixel patches, the Object-Centric Video Transformer (Generative Video Transformer: Can Objects be the Words?) performs unsupervised object discovery to treat individual objects as the discrete "words" of a scene. For long-video representation, the LV-MAE framework decouples short-span and long-span dependencies by treating entire short-video segment embeddings as discrete tokens within a sequence, drastically reducing the computational footprint.
Raw Video ──> [ Segment Encoder ] ──> High-Level Token Sequence ──> [ Long-Range AR Planner ]
│
(Coarse Blueprint)
▼
High-Fidelity Video Output <──────── [ Diffusion Transformer ] ◄─────────────┘
Discrete tokens can also serve as a high-level planning mechanism to guide other generative paradigms, such as diffusion models. Operating directly on raw pixels makes it difficult for diffusion models to maintain long-term consistency, while training them on discrete latents can sometimes limit fine-grained visual quality.
To solve this, Huang et al. demonstrate that compressing frames into a discrete latent space via vector quantization before training a generative model allows it to capture complex temporal dynamics and multi-modal distributions more efficiently.
Taking this hybrid concept further, Li et al. introduce ARLON, which pairs an autoregressive sequence model with a Diffusion Transformer (DiT). The autoregressive model processes a vocabulary of discrete tokens to generate a coarse spatio-temporal "blueprint" of the video. This blueprint then guides the DiT, which handles the high-resolution, continuous details. By using discrete tokens for long-range planning and diffusion for texture generation, the framework achieves consistent, long-video synthesis while significantly reducing inference time.
Key Foundational Work
Video (language) modeling: a baseline for generative models of natural videos (2014) — contributed 1 concept elements to this direction.
Temporal Sequence Modeling for Video Event Detection (2014) — contributed 1 concept elements to this direction.
Video Event Classification Using Bag of Words and String Kernels (2009) — contributed 1 concept elements to this direction.