The dominant paradigm in natural language processing relies on a fragile assumption: that human language can be neatly carved into a fixed, immutable dictionary of tokens before a model ever begins to train. Whether using Byte Pair Encoding (BPE), WordPiece, or Unigram, modern language models map text to a static set of discrete token IDs.
As noted by Deiseroth et al. in their work on T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings, while neural architectures have advanced rapidly, this underlying reliance on predefined, frequency-based vocabularies has remained remarkably static. This vocabulary rigidity creates a fundamental bottleneck. When a model encounters a word, dialect, typo, or script missing from its initial training distribution, its performance degrades rapidly. To overcome this, a growing body of research focuses on bypassing static vocabulary constraints entirely, shifting toward token-free, byte-level, visual, and continuous text representations.
The Out-of-Vocabulary Bottleneck
To understand why static vocabularies fail, we must look at how they handle the open-ended nature of language. A fixed vocabulary forces a hard trade-off between model size and coverage. If a vocabulary is too small, it fractures rare or morphologically complex words into meaningless subword fragments; if it is too large, the embedding and output softmax layers consume a massive, prohibitive share of the model's parameters.
This out-of-vocabulary (OOV) problem is not unique to modern transformers. Early speech recognition systems, such as the Mandarin dictation machine developed by Lee et al., faced severe scalability limits because their predefined dictionaries required manual intervention to incorporate new terminology. Similarly, the classic adaptive language models of Jelinek et al. could adjust the statistical weights of known words in real-time but remained fundamentally bounded by their inability to learn entirely new terms during a session.
In morphologically rich languages, the constraint is even more acute. For instance, Heintz demonstrated that standard word-based models fail to capture the productive nature of Arabic, where new surface forms are constantly generated from core roots. While subword tokenizers were introduced to mitigate this, they remain brittle. In adversarial settings or informal communication, minor variations—such as intentional typos, slang, or "leetspeak"—generate OOV tokens that completely derail standard token-based classifiers (Mou et al.).
Going Token-Free via Bytes and Characters
If the core issue is the discrete tokenization step, the most direct solution is to eliminate the tokenizer altogether. Instead of mapping text to a frequency-derived vocabulary, models can operate directly on the universal, atomic building blocks of digital text: raw UTF-8 bytes or characters.
Operating on raw bytes eliminates the concept of an "unknown" token because every possible character string can be decomposed into a sequence of the same 256 basic byte values. However, raw byte sequences are significantly longer than subword sequences, which increases the computational burden on the self-attention mechanisms of transformers.
To bridge this gap, Sreedhar et al. introduced Local Byte Fusion (LOBEF). Instead of feeding raw bytes directly into a deep transformer, LOBEF uses local convolutions and word-boundary attention to aggregate byte information into dense, localized representations before sequence-level processing. This allows the model to maintain the absolute robustness of a byte-level input space while matching or exceeding the efficiency and translation quality of traditional BPE models.
Standard Pipeline: [Raw Text] -> [Fixed BPE Tokenizer] -> [Static Token IDs] -> [Embedding Matrix]
Byte-Fusion: [Raw Text] -> [Raw UTF-8 Bytes] -> [Local Convolution/Fusion] -> [Dense Vectors]
An alternative approach is to use sparse, hashed character representations. In T-FREE, Deiseroth et al. bypassed subword tokenizers by hashing character sequences into sparse binary vectors. This completely removes the need for a massive token embedding matrix, resulting in huge parameter savings and making the model inherently resilient to cross-lingual shifts and unseen words. Similarly, Wehrmann et al. showed that character-level convolutions can project raw text into shared multimodal spaces, bypassing the fragility that fixed vocabularies exhibit when encountering informal language or typographical errors.
Reading Text as Images: Pixel-Based Models
Perhaps the most radical departure from static vocabularies is the shift toward pixel-based language modeling. Human beings do not read text by processing discrete integer IDs; we look at visual shapes. By rendering text as images and processing the resulting pixels, models can bypass textual tokenization entirely.
This mechanism is incredibly powerful for noisy, non-standard, or historical text. In PHD: Pixel-Based Language Modeling of Historical Documents, Borenstein et al. trained a model to reconstruct masked pixel patches of document scans rather than predicting text tokens. Because it operates on visual features, the model is completely immune to the transcription errors, non-standard historical spellings, and layout anomalies that typically break optical character recognition (OCR) and token-based language models.
Text Input: "Guten Tag" vs. "Guten Tag" (in a regional dialect or old script)
Token-Based: [Token 104] vs. [Token 402, Token 8911, Token 32] (Fractured representation)
Pixel-Based: [Visual Image] -> High structural and morphological overlap preserved
Pixel-based models also solve the problem of dialectal variation. Standard tokenizers frequently fracture regional dialects or non-standardized orthographies into meaningless sub-tokens because those specific letter combinations were rare in the pre-training corpus. Blaschke evaluated pixel language models on non-standardized German dialects, demonstrating that visual language models significantly outperform traditional token-based models like BERT in zero-shot transfer tasks. The visual rendering preserves the structural and morphological similarity between a dialect and the standard language, allowing the model's vision encoder to naturally bridge the linguistic gap.
Dynamic and Continuous Alternatives
Rather than abandoning tokens entirely, some researchers focus on making the output space fluid, moving away from the standard discrete softmax layer over a fixed vocabulary matrix.
One approach is dynamic tokenization. Instead of freezing token boundaries during pre-training, Feher et al. developed a method to retrofit existing large language models, allowing them to dynamically merge subwords into longer, context-specific tokens on-the-fly. By using a hypernetwork to predict embeddings for these newly synthesized tokens, they reduced sequence lengths by over 20% without requiring expensive full-model retraining.
Another paradigm shift involves changing what the model predicts. Traditional neural machine translation models rely on a discrete softmax layer tied to a fixed vocabulary, which creates a severe bottleneck when adapting to low-resource language varieties or new dialects. To break this constraint, Kumar et al. proposed a framework that replaces the discrete softmax layer with a mechanism that predicts continuous word vectors. Because the output space is continuous rather than indexed, new dialects and unseen words can be "plugged in" seamlessly without altering the underlying network architecture.
Finally, the generation process itself can be redefined. In Copy is All You Need, Lan et al. introduced the COG (Copy-Generator) model. Instead of generating text token-by-token from a static internal vocabulary, COG retrieves and "pastes" variable-length phrases directly from a massive external document collection.
Standard Generation: [Context] -> Softmax over 50,000 fixed tokens -> Predict next token
COG Generation: [Context] -> Pointer to external text library -> Copy phrase of variable length
This architecture decouples the model's capacity from a fixed vocabulary matrix: to update the model's terminology, add specialized jargon, or adapt it to a new domain, a developer simply updates the external reference library, completely bypassing the static vocabulary bottleneck.