Large Language Models (LLMs) treat the world as a sequence of discrete symbols. To a Transformer, the word "apple" is an entry in a vocabulary, represented by an arbitrary index like 5021. The problem arises when we ask these models to handle mathematics, physics, or tabular data: to a standard tokenizer, the number "9" is just as unrelated to "8" as it is to "apple."
This representational gap—treating continuous magnitudes as discrete, categorical tokens—is a fundamental bottleneck in applying LLMs to scientific and numerical tasks. Researchers are now developing "numerical inductive biases" to help models understand that numbers exist on a continuum.
The Tokenization Bottleneck
Standard tokenization schemes like Byte Pair Encoding (BPE) are designed for the statistical patterns of human language, not the logic of mathematics. As Boris van Breugel and Mihaela van der Schaar argue in Why Tabular Foundation Models Should Be a Research Priority, these schemes often fragment numbers into arbitrary sub-word units. For example, the number "1234" might be split into "12" and "34".
This fragmentation destroys mathematical continuity. When a model predicts the next digit in a sequence, it isn't performing a calculation; it is performing a classification task across thousands of possible tokens. This leads to several failure modes:
- Precision Loss: Models struggle to maintain the physical meaning of high-precision values, as seen in LaMAGIC.
- Lack of Magnitude Awareness: Models do not inherently know that
9.1is closer to9.2than to100.0. - Poor Extrapolation: As noted in LIFT, models often fail to predict values outside the range of their training data because they haven't learned the underlying function, only the string patterns.
Encoding Numbers as Continuous Values
To fix this, researchers have proposed moving away from string-based representations. A landmark approach is xVal, introduced by Siavash Golkar, Miles Cranmer, and colleagues in xVal: A Continuous Number Encoding for Large Language Models.
Instead of multiple tokens for different digits, xVal uses a single dedicated [NUM] token. The actual numerical value is then used to scale the embedding of that token. This makes the model end-to-end continuous: the input is the value itself, and the model can use standard regression techniques to predict numerical outputs. This approach allows the model to capture the "smoothness" of mathematical functions, leading to better interpolation and scientific reasoning.
Other methods, like those in Vector-ICL, bypass text entirely by using lightweight projectors to map continuous vector representations (from fMRI or time-series data) directly into the LLM's internal embedding space.
Learning Proximity through Loss Functions
If we cannot change the tokenizer, we can change how the model learns. Standard LLMs use cross-entropy loss, which penalizes the model equally whether it guesses "9" when the answer is "8" or when the answer is "1,000,000."
In Regress, Don't Guess, researchers introduced Number Token Loss (NTL). NTL is a proximity-based loss function that forces the model to respect numerical distances. By penalizing the model more heavily for "far away" guesses, the model begins to organize its internal representation of numerical tokens into a coherent number line. Similarly, Laying Anchors suggests "priming" rare numbers by relating them to well-known "anchor" numbers, providing a semantic bridge to magnitude.
Applications in Tabular and Scientific Modeling
The most immediate impact of these techniques is in Large Tabular Models (LTMs). While tree-based models like XGBoost have long dominated tabular data, they lack the transfer learning capabilities of Transformers.
Models like TP-BERTa and TAPTAP attempt to bridge this by combining feature names (text) with their values (numbers). By using specialized numerical embeddings, these models can reason across different table schemas, performing tasks like missing value imputation or synthetic data generation (DP-LLMTGen) with a level of "quantitative commonsense" that standard LLMs lack.
As Charles Zhang and co-authors suggest in their survey on Multimodal Embeddings, the future of LLMs lies in moving beyond "word vectors" toward high-dimensional spaces that can natively represent the continuous nature of the physical world.