To a standard language model, the number 1234 is often treated no differently than the word apple. Most Large Language Models (LLMs) rely on subword tokenization algorithms like Byte Pair Encoding (BPE), which group characters based on their frequency in text corpora. As a result, a common number like 2024 might be assigned a single, atomic token, while an uncommon number like 19847 might be fragmented into arbitrary chunks like 19, 84, and 7.
As argued by Avijit Thawani et al. in their survey Representing Numbers in NLP: a Survey and a Vision, this inconsistent fragmentation obscures the mathematical relationships between digits and creates a severe "modality gap" for numerical data. When numbers are split haphazardly, the model cannot easily perceive their internal structure or relative magnitudes. Eric Wallace et al. demonstrated in Do NLP Models Know Numbers? Probing Numeracy in Embeddings that this leads to an "extrapolation gap," where models fail to reason about numbers even slightly outside their training distribution. Furthermore, Dimitris Spathis et al. note in The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models that this tokenization style fundamentally mangles temporal and sensor data, treating continuous numerical scales as collections of meaningless, discrete text fragments.
The Mechanics of Digit-Level Decomposition
To solve this, researchers have converged on digit-level tokenization (or digit-wise decomposition). Instead of allowing a subword tokenizer to group digits arbitrarily, numbers are explicitly broken down into their fundamental primitives. For example, the number 1234 becomes a sequence of four distinct tokens: 1, 2, 3, and 4. Sometimes, explicit delimiters like spaces or commas are inserted to force this behavior, as highlighted in a survey by Xiyuan Zhang et al. (Large Language Models for Time Series: A Survey).
This granular representation allows the transformer's self-attention mechanism to map operations directly to place-value logic. As shown by Mor Geva et al. in Injecting Numerical Reasoning Skills into Language Models, treating numbers as sequences of individual digit tokens allows the model to capture positional logic and decimal relationships.
For continuous real numbers or data spanning multiple orders of magnitude, this decomposition is often extended to scientific notation or structural components. Deepak Ramachandran et al. introduced NumBERT in Do Language Embeddings capture Scales?, which separates a number into its mantissa and exponent, allowing the model to process precision and magnitude as distinct features. Similarly, for complex mathematical tasks like symbolic regression, Stéphane d'Ascoli et al. in ODEFormer: Symbolic Regression of Dynamical Systems with Transformers and Francois Charton et al. in Learning advanced mathematical computations from examples decomposed floating-point numbers into explicit tokens for the sign, mantissa, and exponent, mapping continuous values into a predictable, discrete sequence.
Why It Works: Data Efficiency and Compositionality
Why does this simple representational shift dramatically improve a model's mathematical capabilities? The intuition lies in two core concepts: vocabulary efficiency and structural exposure.
First, grouping multiple digits into single tokens creates an exponentially large combinatoric space. To learn arithmetic on multi-digit tokens, a model would need to see an unmanageable variety of combinations. By contrast, Haotong Yang et al. point out in Number Cookbook: Number Understanding of Language Models and How to Improve It that single-digit tokenization prevents this exponential vocabulary expansion. The model only needs to master a small, fixed set of numerical primitives (the digits 0–9).
Second, this constraint drastically increases the frequency of these primitives during training. Zhejian Zhou et al. demonstrated in Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia that base-10 (digit-by-digit) tokenization is significantly more data-efficient than multi-digit tokenization (like base-100 or base-1000) because the model encounters individual digits far more frequently. This allows the network to efficiently learn the underlying rules of arithmetic—such as addition and carrying—via standard next-token prediction, a mechanism also leveraged by Anya Sims et al. in StochasTok: Improving Fine-Grained Subword Understanding in LLMs to help models "grok" multi-digit operations. As Weijie Feng et al. observed in GraphMR: Graph Neural Network for Mathematical Reasoning, decomposing numbers into constituent digits and their associated bases allows neural networks to handle an infinite range of numerical values using a small, finite vocabulary.
Generalization and Downstream Precision
By exposing the internal structure of numbers, digit-level representations unlock the ability to generalize to unseen data. Jeonghwan Kim et al. showed in Have You Seen That Number? Investigating Extrapolation in Question Answering Models that decomposing numbers into individual digits alongside explicit place-value tokens enables models to extrapolate to numbers much larger than those seen during training, bridging the traditional extrapolation gap.
This precision is critical across various specialized domains:
- Scientific and Financial Regression: In scientific models like BioT5+, Qizhi Pei et al. found in BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning that character-based digit tokenization prevents inconsistent representations and significantly enhances regression accuracy for predicting continuous physical properties. In finance, Gagan Bhatia et al. utilized a BPE tokenizer that strictly maintains individual digit identity in FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models to preserve granular data precision.
- Time Series and Tabular Data: For time series forecasting, Matthieu Boileau et al. demonstrated in Towards Interpretable Time Series Foundation Models that rescaling data to a fixed integer range and tokenizing digit-by-digit avoids the formatting errors of standard scientific notation, leading to higher precision when identifying extrema. For tabular prediction, Jian Li et al. showed in Generative Table Pre-training Empowers Models for Tabular Prediction that digit-sequence encoding allows generative models to accurately capture continuous and discrete numerical distributions.
- Multimodal Tasks: Even in audio and music processing, Joshua P Gardner et al. found in LLark: A Multimodal Instruction-Following Language Model for Music that tokenizing digits separately prevents precision loss during regression tasks like estimating beats per minute.
The broad utility of this paradigm was confirmed in the NumEval benchmark evaluation by Chung-Chi Chen et al. (SemEval-2024 Task 7: Numeral-Aware Language Understanding and Generation), which showed that decomposing numbers into individual digit tokens consistently improves a model's ability to recognize magnitudes and perform digit-sensitive tasks like rounding or comparison.
Alternative Paradigms and Geometric Nuances
While digit-level tokenization provides clear benefits for arithmetic logic, it alters how numbers are represented internally. Xinyan Hu et al. discovered in Understanding In-context Learning of Addition via Activation Subspaces that models using single-token representations tend to develop cleaner, periodic geometric structures in their activation subspaces for in-context learning of addition, whereas multi-token digit-by-digit representations change the geometric processing of arithmetic logic entirely.
Recognizing the trade-offs of long sequence lengths introduced by digit-by-digit splitting, some researchers explore hybrid approaches. For instance, Tianyi Zhou et al. proposed FoNE in FoNE: Precise Single-Token Number Embeddings via Fourier Features, which maps numbers directly into the embedding space as single tokens using multi-scale Fourier features, but employs a specialized head that decodes outputs digit-by-digit. Alternatively, Xingyuan Ma et al. decoupled numerical values from the text vocabulary entirely in Regression over Classification: Assessing Image Aesthetics via Multimodal Large Language Models by using specialized position placeholders to avoid the structural loss associated with standard text tokenizers.
Key Foundational Work
Injecting Numerical Reasoning Skills into Language Models (2020) — contributed 2 concept elements to this direction.
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019) — contributed 2 concept elements to this direction.
Injecting Numerical Reasoning Skills into Language Models (2020) — contributed 1 concept elements to this direction.