CUTE: Measuring LLMs' Understanding of Their Tokens

Lukas Edman, Helmut Schmid, Alexander Fraser

Abstract

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.

Introduction

Modern Large Language Models (LLMs) do not perceive text as a sequence of individual characters. Instead, they utilize Byte Pair Encoding (BPE) to group characters into atomic units known as tokens. While this approach is computationally efficient, it creates a potential "blind spot": the model processes the word "apple" as a single unit rather than the letters 'a', 'p', 'p', 'l', and 'e'. This architectural choice raises a fundamental question: Do LLMs truly understand the internal structure of their tokens, or are they merely reciting memorized patterns?

The inability to "see" inside tokens often manifests as difficulty in tasks requiring fine-grained character awareness, such as solving crosswords, generating specific rhymes, or deciphering simple codes. This paper introduces the CUTE (Character-level Understanding of Tokens Evaluation) benchmark to systematically probe the limits of LLMs' orthographic reasoning.

The CUTE Benchmark

To isolate character-level knowledge from general linguistic reasoning, the CUTE benchmark consists of tasks that are trivial for humans but challenging for token-based models. The benchmark is structured into three primary categories:

  1. Composition: Evaluates basic knowledge, such as spelling a word (Word to Letters), reverse spelling, and identifying if a specific character exists within a token.
  2. Similarity: Tests whether a model can distinguish between semantic similarity (meaning-based, e.g., happy vs. glad) and orthographic similarity (visual/spelling-based, e.g., happy vs. apply).
  3. Manipulation: Requires the model to perform edits like insertion, deletion, substitution, and swapping. These are tested at both the character level and the word level to differentiate between instruction-following ability and character-level visibility.
Overview of the CUTE benchmark tasks across Composition, Similarity, and Manipulation categories.

Key Findings

Shallow Memorization vs. Functional Understanding

The evaluation of popular open-weight models (including Llama 2/3, Mistral, and Command-R) revealed a striking discrepancy. While models are generally proficient at spelling out common words (Composition), their performance plummets when asked to manipulate those same characters. This suggests that spelling is stored as a "shallow" secondary attribute of a token—a memorized sequence—rather than a functional understanding of the word's internal structure.

The "Vision" Gap

A critical insight from the benchmark is that models perform significantly better at word-level manipulation than character-level manipulation. For instance, in the "Insertion" task, Command-R+ exhibited a 72.8% accuracy gap between word-level and character-level versions of the same instruction. This confirms that the models understand the logic of the task (e.g., "insert X after Y") but lack the "vision" to locate the correct insertion points within a token.

Orthographic Blindness

Perhaps the most telling result is the models' performance on similarity tasks. While models excel at identifying words with similar meanings, they are nearly "blind" to visual similarity. Most models performed near the random chance baseline when asked to identify which words were orthographically similar, even though they were highly accurate at semantic matching.

Comparison of model performance on Semantic vs. Orthographic similarity tasks, showing a significant gap in visual reasoning.

Tokenization as the Primary Barrier

To confirm that tokenization is the root cause, researchers tested models on random consonant strings (e.g., "fxqg"). In these cases, the tokenizer is forced to treat almost every character as an individual token. Interestingly, model performance improved on these strings, proving that when the "one character = one token" mapping is restored, the models' character-level intelligence increases.

Conclusion

The CUTE benchmark highlights a fundamental architectural limitation in current LLMs. While scaling models to larger parameter counts (e.g., 70B+) provides some improvements, it does not solve the underlying "orthographic blindness" caused by BPE tokenization. Future research into native character-level models or hybrid tokenization strategies may be necessary to bridge this gap and enable LLMs to achieve true human-level proficiency in character-level reasoning.

Summary updated

Abstract

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.
BibTeX
@inproceedings{edman2024cute,
  author = {Lukas Edman and Helmut Schmid and Alexander Fraser},
  title = {{CUTE: Measuring LLMs' Understanding of Their Tokens}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=PbuIs33Qhk},
}
Figure