To a human reader, the word "strawberry" is clearly composed of ten letters, including three 'r's. But to a state-of-the-art large language model (LLM), "strawberry" is not a sequence of letters at all. Instead, it is ingested as an atomic integer—a token ID.
This design choice stems from subword tokenization algorithms like Byte-Pair Encoding (BPE) or WordPiece. While tokenization allows models to efficiently compress text and handle massive vocabularies, it introduces a fundamental architectural blind spot: character-level opacity. As Zhu Xu et al. point out, treating text segments as atomic units prevents models from directly accessing the internal orthographic structure of words. This explains why an LLM can pass a bar exam but struggle to count the letters in a simple word.
Why Tokenization Creates a Leaky Abstraction
When text is tokenized, the rich structural connection between individual characters is severed. A token ID is just an index in a lookup table. The model's embedding layer maps this ID to a dense vector, but nothing in that initial vector explicitly signals that the token contains specific characters in a specific order.
As Yidan Zhang et al. demonstrate, this creates a structural barrier for tasks requiring micro-level linguistic awareness, such as string reversal or letter counting. Crucially, Chai et al. show that this "character blindness" cannot be solved simply by scaling up the model's parameters or training data. Even massive models hit a performance wall as word length increases because the disconnect between token IDs and their constituent characters is an architectural limitation, not a capacity issue. Vieira et al. describe the tokenization layer as a "leaky abstraction" that obscures accurate character-level probabilities and constraints, making it incredibly difficult for models to satisfy strict character-level boundaries.
Probing the Hidden Orthography
If the input representation is completely opaque, how do models manage to perform any character-level tasks at all? For instance, Rozner et al. observed that subword-based models are surprisingly capable of solving certain wordplay puzzles like anagrams, suggesting that the grouping of characters doesn't completely block access to the underlying sequences.
To investigate this paradox, Itzhak et al. used a generative probe called SpellingBee and discovered that LLMs implicitly recover and store a significant amount of orthographic information within their embeddings during standard next-token prediction training. Furthermore, Nan Xu et al. argue that character-level failures are often failures of execution rather than capability; when models are prompted to reason step-by-step before responding, they can successfully navigate internal word structures that they otherwise seem blind to.
Injecting Character Awareness into Subwords
To bridge the gap between subword efficiency and character-level precision, researchers have developed methods to force models to learn internal token structures without abandoning the efficiency of a vocabulary.
One approach is stochastic tokenization. Instead of always splitting a word into the exact same BPE tokens, Sims et al. introduce StochasTok, which randomly breaks tokens into smaller sub-units during training. This forces the model to learn the relationships between different granularities of the same word, exposing its internal structure.
Another approach is internal surgery. Huang et al. propose Type-level Interchange Intervention Training (IIT), a causal intervention method that explicitly trains subword-based models to align their internal representations with the exact character composition of their tokens. This results in models that retain high-level contextual understanding while gaining fine-grained orthographic control.
Tokenizer-Free Architectures
Rather than patching the flaws of subword tokenization, a growing faction of research advocates for eliminating tokenizers entirely, processing raw bytes directly.
Historically, byte-level models were too computationally expensive because text sequences became four to five times longer, causing the quadratic cost of self-attention to explode. However, the Byte Latent Transformer (BLT) proposed by Pagnoni et al. bypasses this by dynamically grouping raw bytes into "patches" based on data entropy. This allows the model to maintain the efficiency of token-based models while enjoying full character-level visibility and robustness to noise. Similarly, Deng et al. introduce ByteFlow Net, which uses an information-theoretic objective to compress raw byte streams adaptively on the fly, ensuring that the model never loses access to the underlying character structure.
Real-World Impacts: Visual Text and Multilinguality
The consequences of character blindness extend far beyond word games and spelling bees; they directly impact multimodal systems and multilingual applications.
In text-to-image generation, models frequently output garbled text or spelling errors. Norouzi et al. demonstrate that replacing standard "character-blind" text encoders (like CLIP or T5) with character-aware alternatives dramatically improves visual spelling accuracy. For instance, Glyph-ByT5, developed by Zeyu Liu et al., uses a character-aware, glyph-aligned encoder to boost visual text rendering accuracy from under 20% to nearly 90%. Additionally, the authors of GlyphControl use explicit glyph images as spatial blueprints to overcome the limitations of BPE-encoded text inputs.
In the multilingual domain, Dussolle et al. show that subword tokenization acts as a severe bottleneck for languages with complex script systems or lower data representation, where models frequently fail to follow precise character or script constraints. By moving toward character-aware tokenization or tokenizer-free architectures, models can achieve the fine-grained accuracy required for optical character recognition, precise translation, and complex script manipulation.
Key Foundational Work
Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT (2020) — contributed 1 concept elements to this direction.
Named Entity Extraction Based on A Maximum Entropy Model and Transformation Rules (2000) — contributed 1 concept elements to this direction.