Large Language Models (LLMs) process the world through the lens of discrete tokens. While this discrete framing works exceptionally well for languages, where words and sub-words form distinct categorical units, it creates a fundamental mismatch when applied to the continuous physical and mathematical world.
When an autoregressive transformer encounters a number like 3.1415, it does not see a point on a continuous number line. Instead, standard tokenizers fragment this value into a sequence of arbitrary sub-word strings or digits (e.g., ["3", ".", "14", "15"]). This discrete tokenization scheme strips away the mathematical relationships inherent to continuous data, leading to a phenomenon known as numerical magnitude blindness.
The Tokenization Barrier and Magnitude Blindness
To understand why standard language models struggle with continuous values, we have to look at how they represent vocabulary. In a standard transformer, every token in the vocabulary corresponds to an independent vector in an embedding lookup table. The model treats these tokens as categorical entities. Under standard cross-entropy loss, predicting the token "4" when the true target is "5" is penalized just as heavily as predicting "100" or "banana".
As Eric Wallace et al. observed, treating numbers as discrete symbols prevents models from understanding the continuous nature of the number line, severely limiting their ability to extrapolate beyond the exact numerical ranges seen during training. This lack of inductive bias manifests as a series of critical failures across scientific and structured data domains:
- Tabular Data Inefficiency: Standard autoregressive models struggle to maintain statistical calibration or capture underlying continuous distributions in tables because digit-by-digit tokenization obscures the relative scales between data points, as argued by Boris van Breugel et al. and Zhenhan Huang et al..
- Sensitivity to Noise: In time-series forecasting, minor data distortions can completely scramble a model's predictions. Junwoo Park et al. demonstrated that a tiny numerical shift (e.g., from
0.99to1.00) can result in drastically different token sequences, breaking the model's ability to recognize continuous mathematical patterns. - Geometric Blindness: When processing 3D spatial coordinates or geographic data, models process numbers as text strings rather than continuous vectors. Nawaf Alampara et al. and Wenbin Li et al. showed that this prevents models from performing precise geometric calculations or understanding spatial proximity, often causing them to snap to a rigid grid of previously seen values.
Embedding Magnitude Directly
To bridge this gap, researchers have developed methods that inject continuity directly into the transformer's embedding space. The core intuition is simple: instead of mapping different numbers to entirely different token slots, we can use the scalar value of the number to scale a shared embedding vector.
A prominent example of this paradigm is xVal, introduced by Siavash Golkar et al.. Rather than tokenizing a number into individual digits, xVal represents every continuous number using a single, generic [NUM] token. The embedding of this token is then multiplied by the actual scalar value of the number:
$$\mathbf{e}(x) = x \cdot \mathbf{v}_{[NUM]}$$
Where $x$ is the raw numerical value and $\mathbf{v}_{[NUM]}$ is the learnable embedding vector for the number token.
Standard Tokenization: "3.14" --> ["3", ".", "14"] --> [Vector_3, Vector_Dot, Vector_14]
Continuous Encoding: "3.14" --> [NUM] --> 3.14 * Vector_NUM
This approach fundamentally alters how the transformer perceives numbers. Because the input representation scales linearly with the actual value, the self-attention mechanism can naturally compute smooth functions and interpolate across a continuous range. Gradients can flow directly from the numerical values back into the network, allowing the model to recognize that values close in magnitude should behave similarly.
Other architectural variations achieve a similar effect by decoupling the numerical value from the text stream. For instance, in visual and spatial reasoning tasks, Peter Kulits et al. bypassed text tokenization entirely by appending a specialized "numeric head" to the language model, allowing it to output precise, continuous spatial coordinates directly via regression rather than text generation.
Teaching Metric Distance to Loss Functions
Modifying the input embeddings solves only half of the problem; the standard training objective remains metric-blind. When an autoregressive model predicts the next token, the final layer outputs a softmax distribution over discrete vocabulary classes. This creates a non-differentiable barrier between the model's output and the actual mathematical magnitude of the error.
To fix this, researchers have redesigned loss functions to penalize mistakes based on how far apart the predicted and true values are on the real number line.
Target: "5"
Prediction A: "4" --> Standard Cross-Entropy: WRONG (Full Penalty) --> Distance Loss: Small Penalty
Prediction B: "9" --> Standard Cross-Entropy: WRONG (Full Penalty) --> Distance Loss: Large Penalty
Jiwan Chung et al. introduced DIST$^2$Loss, a distance-aware objective function that incorporates the metric relationships between tokens directly into the loss calculation. Instead of treating all token mismatches equally, the loss scales with the physical or mathematical distance between the predicted category and the target category.
Similarly, Xiang Fei et al. developed Numerical Token Integrity Loss (NTIL). NTIL leverages the Earth Mover’s Distance (Wasserstein metric) to optimize sequence-level numerical predictions. By shifting the optimization target from discrete classification to optimal transport along the number line, the loss ensures that the model inherently understands that 0.98 is mathematically close to 1.01, even if their string representations share no common tokens.
By aligning both the input representations and the optimization objectives with the topology of the real number line, these techniques allow autoregressive architectures to shed their categorical constraints, transforming them from simple text pattern-matchers into continuous mathematical reasoners.
Key Foundational Work
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers (2018) — contributed 2 concept elements to this direction.
Numerically Grounded Language Models for Semantic Error Correction (2016) — contributed 1 concept elements to this direction.
Combining Symbolic Expressions and Black-box Function Evaluations in Neural Programs (2018) — contributed 1 concept elements to this direction.