When humans look at the numbers 10, 11, and 1,000, we instantly recognize a geometric reality: 10 and 11 are nearly identical, while 1,000 is orders of magnitude larger. Neural networks trained on text, however, see these numbers through the distorting lens of subword tokenization. A standard tokenizer might split "1024" into tokens like ["10", "24"], completely severing the mathematical connection between adjacent values. To a network, the token for "10" is no closer to "11" than it is to "banana."
This "number blindness" severely limits neural networks in scientific computing, financial forecasting, and tabular data processing. To solve this, a prominent research direction focuses on magnitude-aware numerical encodings—methods that map continuous numbers into vector spaces where geometric distance directly reflects mathematical relationships.
The Limits of Tokenization
In standard language models, numbers are treated as discrete vocabulary words. As surveyed by Ao Liu et al., this fragmentation degrades numerical data, stripping away magnitude, precision, and rank. Without an explicit representation of scale, models struggle to perform basic arithmetic or recognize ordinal constraints (such as identifying the "largest" or "smallest" entity in a dataset).
To build true numeracy, a model's internal representations must preserve both cardinality (the absolute size of a value) and ordinality (the relative position on a number line), as emphasized by Leilei Gan et al..
Continuous Projections and Scalar Scaling
The most direct way to preserve magnitude is to bypass text tokenization entirely and inject raw numerical scalars into the network. Early foundational work by Georgios P. Spithourakis et al. demonstrated this by feeding continuous floating-point scalars directly into a language model's hidden states, allowing the network to interpret physical magnitudes natively.
For modern Transformer architectures, this concept has evolved into scaling dedicated placeholder tokens. A prominent example is xVal, introduced by Siavash Golkar et al., which represents any number using a single generic [NUM] token. Instead of changing the token based on the digits, the model multiplies the token's embedding vector by the actual scalar value of the number. This ensures that the embedding space responds smoothly and continuously to changes in magnitude.
Standard Tokenizer: "1729" --> [17] [29] (Discrete, fragmented)
Continuous Scaling: "1729" --> 1729 × [NUM] (Smooth, magnitude-preserving)
Similar direct projection strategies have proven vital across diverse domains:
- Circuit Design: Chen-Chia Chang et al. project continuous circuit specifications directly into a model's embedding space via a dedicated linear layer, preventing the mathematical dilution caused by text tokenizers.
- Clinical Video Analysis: To help vision-language models understand quantitative metrics, Diwei Wang et al. scale a dedicated base vector by the specific numerical value, ensuring the model recognizes the proximity and scale of clinical measurements.
Periodic and Fourier Feature Mappings
While direct linear scaling preserves absolute magnitude, it introduces a major bottleneck: neural networks inherently struggle to approximate highly non-linear, jagged, or high-frequency functions when given a single raw scalar input. This limitation is why simple multi-layer perceptrons (MLPs) historically underperformed compared to tree-based models (like XGBoost) on tabular datasets.
To overcome this, researchers use periodic encodings. By passing a scalar value through a series of sine and cosine functions at varying frequencies, a single number is mapped into a rich, high-dimensional vector. This functions exactly like a positional encoding for the continuous number line.
$$\text{Embedding}(x) = [\sin(\omega_1 x), \cos(\omega_1 x), \sin(\omega_2 x), \cos(\omega_2 x), \dots]$$
This periodic transformation allows standard neural networks to capture complex, non-linear relationships with ease. David Holzmüller et al. demonstrated that equipping standard MLPs with periodic embeddings (RealMLP) allows them to match the performance of gradient-boosted decision trees on tabular data. In a similar vein, Ivan Rubachev et al. showed that periodic numerical encodings are the critical ingredient enabling simple architectures to capture patterns that standard MLPs miss entirely.
Furthermore, mapping numbers to multi-scale Fourier features allows networks to generalize mathematical operations perfectly. Tianyi Zhou et al. introduced FoNE, showing that Fourier-based single-token embeddings enable small language models to achieve 100% arithmetic accuracy while requiring significantly less training data than subword tokenization models.
Scalar Input (x) ───► [ Sinusoidal / Fourier Filters ] ───► High-Dimensional Vector
(Multi-scale frequencies) (Captures fine & coarse patterns)
This periodic strategy is highly versatile, appearing in heterogeneous transactional time-series models to preserve precision without discretization (Fabrizio Garuti et al. et al.), healthcare risk prediction models to transform real-valued lab results into structured features (Sundreen Asad Kamal et al.), and tabular sequence models like Mambular (Anton Frederik Thielmann et al.).
Distributional and Relational Encodings
In some scenarios, absolute magnitude matters less than relative distribution or ordinal rank. For instance, datasets with extreme outliers can destabilize continuous embeddings.
To handle varying scales robustly, Zhenhan Huang et al. proposed mapping scalars to high-dimensional vectors using a piece-wise linear approach based on quantiles. This normalizes values into specific ranges based on their distribution, providing an inductive bias that protects the model from outlier distortion.
When reasoning over structured networks like Knowledge Graphs, numbers must be integrated alongside discrete entities. Xianshu Peng et al. designed a framework where numerical values are projected into a high-dimensional space where the geometric distance between embeddings corresponds directly to their numerical difference, enabling the model to answer complex statistical and range queries (e.g., "greater than" or "equal to").
Alternatively, Hanyu Duan et al. showed that instead of mapping numbers to absolute scalars, encoding a number's relative position within a structured network line allows models to learn numeracy through comparative logic. This aligns numerical order with the vector space properties of standard word embeddings, proving that the geometry of the number line can be learned relationally.
Summary
Whether through direct scalar scaling, multi-scale Fourier features, or quantile-based binning, magnitude-aware numerical encodings fundamentally change how neural networks see mathematics. By ensuring that the vector space mirrors the continuity and scale of the real number line, these techniques bridge the gap between symbolic literacy and quantitative numeracy.
Key Foundational Work
Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network (2019) — contributed 1 concept elements to this direction.
Clinical Text Prediction with Numerically Grounded Conditional Language Models (2016) — contributed 1 concept elements to this direction.