Abstract
Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.
Introduction
Modern Large Language Models (LLMs) are incredibly fluent, yet they process text in a way that is fundamentally different from human reading. Instead of perceiving individual letters, they use Byte-Pair Encoding (BPE) to break text into subword chunks called "tokens." For instance, the word "apple" might be processed as a single atomic ID. This architectural choice leads to a phenomenon known as "orthographic blindness." If a model never explicitly "sees" the letters during its primary processing, does it truly understand the internal structure of its vocabulary?
Character-level understanding is not merely a niche requirement; it is essential for solving word puzzles, writing constrained poetry (such as alliteration), parsing ciphers, and handling morphological inflections. This paper introduces the CUTE (Character-level Understanding of Tokens Evaluation) benchmark to investigate the gap between a model's ability to predict the next token and its ability to understand the characters that comprise those tokens.
The CUTE Benchmark
The CUTE benchmark is designed to evaluate three specific dimensions of orthographic knowledge:
- Composition: Testing if a model can spell a word forward and backward, or identify if a specific character is present within a token.
- Similarity: Determining if a model can distinguish between semantic similarity (meaning) and orthographic similarity (spelling). For example, recognizing that "happy" is orthographically similar to "apply" but semantically similar to "glad."
- Manipulation: Assessing the model's ability to perform character-level "edit" operations, such as insertion, deletion, substitution, and swapping.
To ensure the benchmark measures character knowledge rather than general reasoning or instruction-following abilities, the researchers implemented a word-level control for every task. For example, a character-level task like "Add 'b' after every 'e' in 'there'" is paired with a word-level control like "Add 'is' after every 'the' in 'the sky is blue'." A performance gap between these two tasks highlights a specific deficiency in orthographic processing.
Methodology and Datasets
The benchmark utilizes several data sources to ensure robustness:
- CUTE (English): Synthetically generated tasks using frequent English words from the Google Web Trillion Word Corpus to ensure the words are likely represented as single tokens.
- TinyStories: Used to construct word-level control tasks with simple, consistent sentence structures.
- CUTE-Rus: A Russian adaptation of the benchmark designed to test if orthographic blindness persists across different scripts and multilingual models.
Key Insights
The Spelling Bee Paradox
Evaluation results reveal a striking contradiction: most models are highly proficient at spelling words out loud (often exceeding 90% accuracy). However, their performance plummets when they are required to manipulate those same letters, such as swapping characters within a word. This suggests that models have "memorized" spelling as a static property of a token rather than understanding the word as a dynamic sequence of characters.
Tokenization as a Bottleneck
A critical experiment involved testing models on random strings of consonants. Because these strings do not form common words, the tokenizer is forced to break them into individual characters. Interestingly, model performance improved on these random strings compared to real words. This confirms that the "atomic" nature of subword tokens is a primary hindrance to character-level logic.
The Limits of Scaling
While larger models generally perform better, scaling parameters does not bridge the gap between word-level and character-level manipulation. For example, Command-R+ exhibited a 72.8% performance gap between its success at the word level versus the character level on insertion tasks. This indicates that orthographic understanding is an architectural or training data issue rather than a simple matter of model size.
Conclusion
The CUTE benchmark demonstrates that current LLMs suffer from significant orthographic blindness. While they can recall spellings from memory, they struggle to apply that knowledge to functional tasks. These findings suggest that for LLMs to master complex linguistic tasks like poetry or advanced coding, researchers may need to explore character-level architectures (such as ByT5) or develop training objectives that encourage models to internalize the internal structure of their tokens.
Summary updated