When large language models process natural language, they rely on subword tokenization algorithms like Byte Pair Encoding (BPE) or WordPiece. These algorithms are designed to maximize compression by grouping frequently co-occurring characters into single tokens. While this works exceptionally well for words—where prefixes, suffixes, and roots repeat predictably—it introduces a fundamental structural flaw when applied to numbers.
Because tokenizers are optimized for linguistic frequency rather than mathematical logic, they slice multi-digit numbers into arbitrary, non-semantic fragments. For example, the number 2024 might be treated as a single token because it appears frequently in the training corpus, while 2029 might be split into [20, 29]. This phenomenon, known as subword-induced numerical fragmentation, creates a severe bottleneck for numeracy in artificial intelligence.
The Broken Internal Number Line
Humans understand numbers as points along a continuous magnitude scale. We inherently know that $9.11$ is less than $9.9$, and that $1023$ sits directly adjacent to $1024$. Transformers, however, view tokens as discrete, initially orthogonal vectors in a high-dimensional space. They must learn the relationships between these vectors entirely from context.
When subword tokenizers fragment numbers based on string frequency rather than mathematical value, they destroy the ordinal relationships between numbers. As demonstrated by Eric Wallace et al. in Do NLP Models Know Numbers? Probing Numeracy in Embeddings, this fragmentation creates a severe "extrapolation gap." Models can often interpolate numerical relationships within narrow ranges seen during training, but they completely fail when encountering values that are tokenized differently.
Avijit Thawani et al. note in Representing Numbers in NLP: a Survey and a Vision that this fragmentation prevents models from naturally mapping numerically similar values to nearby regions in vector space. Instead of executing mathematical operations, the model is forced to dedicate significant internal parameter capacity simply to reconstructing the original numerical values from disjointed tokens before any actual computation can occur.
The Collapse of Place-Value Logic
To perform basic arithmetic, an agent must understand place-value logic—the principle that the value of a digit depends on its position (ones, tens, hundreds, etc.).
Standard Math Alignment:
1 2 3 4
+ 5 6 7
---------
Arbitrary Subword Fragmentation:
[12] [34]
+ [567]
--------- <-- Alignment is broken; tokens do not correspond to place values.
When a tokenizer aggregates digits into arbitrary clusters, it obscures individual digits and their relative positions. Haotong Yang et al. explain in Number Cookbook: Number Understanding of Language Models and How to Improve It that multi-digit tokenizers create a significant barrier to arithmetic because the relationship between tokens and numerical values becomes completely opaque.
Without a granular, consistent representation of each digit's position, neural networks cannot reliably internalize the rules of mathematical operations. As a result, works like Injecting Numerical Reasoning Skills into Language Models by Mor Geva et al. and Have You Seen That Number? Investigating Extrapolation in Question Answering Models by Jeonghwan Kim et al. emphasize that standard tokenization forces models to rely on memorized statistical patterns of digit sequences rather than a robust understanding of numerical scale and arithmetic rules.
Generation Failures and Probability Leakage
Numerical fragmentation does not just disrupt how models read numbers; it fundamentally breaks how they write them.
In autoregressive language models, text generation occurs token by token. James Requeima et al. point out in LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language that standard tokenization causes "probability mass leakage" between numbers that share the same prefix (such as 12 and 120). Because the model evaluates tokens independently rather than as unified mathematical values, the boundary between distinct numbers becomes blurred.
This structural blind spot frequently leads to digit hallucinations or omissions during sequence reconstruction. As Jun Wang et al. demonstrate in As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation, algorithms like BPE break long numbers into arbitrary chunks that fail to map to mathematical magnitudes. When the model attempts to regenerate these fragments, it often adds, omits, or substitutes significant figures because it lacks a holistic, sequence-level view of the total value.
Downstream Modality Collapses
The consequences of numerical fragmentation extend far beyond simple arithmetic word problems. Because numbers serve as the foundational scaffolding for structured data, time, and space, tokenization bottlenecks cause systemic failures across multiple domains:
- Tabular Data: In surveys like Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey by Xi Fang et al. and Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks by Ao Liu et al., researchers note that splitting multi-digit numbers into non-mathematical units degrades a model's ability to perform calculations, comparisons, or formula predictions within a data grid.
- Time-Series and Sensor Data: Dimitris Spathis et al. show in The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models that standard tokenizers often merge mathematical symbols—such as negative signs—with preceding delimiters like commas. This removes essential directional information from sensor readings, making it impossible for the model to distinguish between positive and negative values.
- Spatial and Geometric Coordinates: In vision-language models, bounding boxes are represented as numerical coordinates. Kesen Zhao et al. argue in Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought that fragmentation destroys the semantic integrity of these coordinates. Similarly, Daniel Franzen demonstrates in Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective that multi-digit token aggregation disrupts the precise coordinate-based alignments required to solve visual reasoning tasks like the Abstraction and Reasoning Corpus (ARC).
- Temporal Understanding: The authors of DateLogicQA: Benchmarking Temporal Biases in Large Language Models found that tokenizers fragment compressed date strings (e.g., dates written without explicit separators) into arbitrary units, destroying the internal semantic structure required to parse days, months, or years.
Emerging Solutions
To bypass the tokenization bottleneck, researchers are exploring architectures that preserve the integrity of numerical data.
Stochastic Tokenization
One immediate mitigation strategy involves altering how models see numbers during training without changing the underlying vocabulary. The authors of StochasTok: Improving Fine-Grained Subword Understanding in LLMs introduce a stochastic tokenization scheme that randomly splits multi-digit clusters into smaller, equivalent sub-units during training. This forces the model to learn the internal structure of numbers and successfully enables them to "grok" multi-digit addition.
Continuous and Fourier Embeddings
Rather than forcing continuous numbers into discrete text tokens, some approaches bypass tokenization entirely. Tianyi Zhou et al. developed FoNE (FoNE: Precise Single-Token Number Embeddings via Fourier Features), which maps numbers directly into the embedding space using multi-scale Fourier features. By representing numbers as single tokens embedded with explicit mathematical properties, even small models can achieve near-perfect arithmetic accuracy while using significantly less training data.
Value-Aware Loss Functions
Other approaches modify the model's optimization target to account for numerical distance. Xiang Fei et al. introduce Numerical Token Integrity Loss (NTIL) in Advancing Sequential Numerical Prediction in Autoregressive Models. Instead of evaluating fragmented tokens independently using standard cross-entropy, NTIL uses sequence-level optimization to ensure the model optimizes for the overall mathematical accuracy of the reconstructed number, teaching the network that 0.98 is conceptually close to 1.01.
Ultimately, as Chung-Chi Chen concludes in SemEval-2024 Task 7: Numeral-Aware Language Understanding and Generation, treating numbers as arbitrary strings remains a fundamental limitation. Moving past this bottleneck requires tokenization and embedding strategies that explicitly respect the unique structural, positional, and quantitative properties of digits.
Key Foundational Work
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019) — contributed 2 concept elements to this direction.
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019) — contributed 2 concept elements to this direction.