When we think about building an AI that can see, hear, speak, and act, the dominant paradigm has long been modular. We take a pre-trained vision encoder (like CLIP), a speech encoder (like Whisper), and stitch them onto a Large Language Model (LLM) using linear layers or cross-attention adapters. While effective, this approach creates an architectural "modality gap." The model treats language as its native tongue, while images and audio are treated as foreign inputs translated through a bottleneck. Furthermore, generating non-textual outputs often requires handing off representations to entirely separate networks, such as diffusion decoders.
A rapidly growing body of research proposes a more elegant alternative: Unified Multimodal Tokenization. The core insight is that a Transformer is fundamentally agnostic to the meaning of its inputs. It does not know if a token represents the word "apple," a patch of a cat's fur, a specific audio frequency, or a robotic joint command. It simply models conditional probabilities over a discrete vocabulary.
By mapping all modalities into a single, shared discrete token space, we can train a single sequence-to-sequence architecture to process and generate any modality natively using standard next-token prediction or masked modeling. As Jun Zhan demonstrates with AnyGPT, large language models possess a fundamental capacity for modality-agnostic reasoning when diverse data types are formatted as discrete semantic tokens.
Discretizing the Indiscretizable
To make a unified vocabulary work, we must first answer a difficult question: How do we convert continuous, high-dimensional data—like the pixels of an image, the waveforms of audio, or the physical trajectories of a robot—into discrete symbols?
Images and Video as Visual Words
For visual data, researchers leverage Vector Quantization (VQ) techniques, such as VQ-VAE or VQGAN. An image is divided into a grid of patches. Each patch is passed through an encoder to produce a continuous vector, which is then mapped to the closest vector in a learned, finite codebook. The index of that codebook vector becomes the token.
David Mizrahi et al. utilize this approach in 4M: Massively Multimodal Masked Modeling, demonstrating that dense geometric maps, text, and neural feature maps can all be co-discretized into a single sequence. Similarly, Shiqi Yang et al. show in Visual Echoes that mapping both audio and video into a shared VQGAN space allows a lightweight transformer to perform bidirectional audio-visual generation without autoregressive overhead.
Actions as Language
Physical actions and continuous control signals can be tokenized through uniform binning or quantization. In their foundational work on A Generalist Agent, Scott E. Reed et al. serialize environment observations, text instructions, and continuous control values into a single token stream. By framing sequential decision-making as standard sequence modeling, a single set of weights can seamlessly switch between playing Atari games, controlling a robot arm, and chatting with a user.
Structured Data
Even highly structured scientific data can be brought into the fold. For instance, Qizhi Pei et al. introduce 3D-MolT5, which maps 3D molecular substructures to discrete tokens. This allows a T5 language model to natively perform cross-modal tasks, such as translating between 3D molecular geometries, 1D chemical strings (SMILES), and natural language descriptions within a single vocabulary.
The Architecture of a Shared Vocabulary
Once every modality is tokenized, they must coexist within a single global vocabulary without causing semantic interference.
Shared Vocabulary Layout:
[ Text Tokens (0-32000) ] [ Visual Codebook (32001-40000) ] [ Action Bins (40001-41000) ]
In the seminal work OFA, Peng Wang designs a unified sequence-to-sequence framework where text tokens, quantized image patches, and discretized spatial bounding box coordinates are integrated into a single representational space. Because every target is a discrete token, the entire training objective simplifies to a standard cross-entropy loss across all modalities.
To maintain structural clarity when mixing modalities, frameworks often employ explicit formatting or prefixing strategies. For example, Jianyang Zhai et al. demonstrate that assigning unique prefixes to tokens based on their source modality allows a generative recommendation model to distinguish between textual descriptions and visual item attributes while sharing behavioral patterns across different product categories.
This serialization enables extreme flexibility during inference. Because the model views all conditions as a continuous stream of tokens, it can handle arbitrary combinations of inputs without needing task-specific architectural adjustments. As Bowen Zhou et al. demonstrate in Context-Aware Autoregressive Models for Multi-Conditional Image Generation, concatenating diverse spatial conditions (like depth maps or Canny edges) and semantic prompts into a single token sequence allows an autoregressive model to achieve zero-shot flexibility, handling novel combinations of constraints natively.
Why It Works: Cross-Modal Synergy
The true power of unified tokenization lies in how it changes the way a model learns. In an adapter-based system, the text model's weights are largely frozen or insulated from the visual features. In a unified tokenization model, every gradient step updates the shared attention layers' understanding of how tokens relate to one another, regardless of their origin.
When a model is trained on a unified sequence, it learns the joint distribution across all modalities simultaneously. Jiasen Lu et al. demonstrate this holistic integration with Unified-IO 2, the first autoregressive model to natively ingest and generate text, images, audio, and actions within a single transformer. By training from scratch on a unified tokenization scheme, the model achieves superior cross-modal reasoning—such as generating audio directly from image tokens or predicting robotic actions from video frames—proving that a single generalist architecture can match or exceed the performance of highly specialized, modular systems.
By reducing the barriers between sight, sound, text, and action to a common mathematical currency—the discrete token—unified tokenization provides a clean, scalable roadmap toward true general-purpose foundation agents.