Continuous Number Representations for Language Models

This direction is split into other directions: discrete numerical representation, digit-level tokenization, numerical tokenization fragmentation. Its last version stays readable and citable here.

Version 1 of 1342 papers489 conceptsfrozen 2026-10-04

Large Language Models (LLMs) treat the world as a sequence of discrete symbols. To a Transformer, the word "apple" is an entry in a vocabulary, represented by an arbitrary index like 5021. The problem arises when we ask these models to handle mathematics, physics, or tabular data: to a standard tokenizer, the number "9" is just as unrelated to "8" as it is to "apple."

This representational gap—treating continuous magnitudes as discrete, categorical tokens—is a fundamental bottleneck in applying LLMs to scientific and numerical tasks. Researchers are now developing "numerical inductive biases" to help models understand that numbers exist on a continuum.

The Tokenization Bottleneck

Standard tokenization schemes like Byte Pair Encoding (BPE) are designed for the statistical patterns of human language, not the logic of mathematics. As Boris van Breugel and Mihaela van der Schaar argue in Why Tabular Foundation Models Should Be a Research Priority, these schemes often fragment numbers into arbitrary sub-word units. For example, the number "1234" might be split into "12" and "34".

This fragmentation destroys mathematical continuity. When a model predicts the next digit in a sequence, it isn't performing a calculation; it is performing a classification task across thousands of possible tokens. This leads to several failure modes:

  1. Precision Loss: Models struggle to maintain the physical meaning of high-precision values, as seen in LaMAGIC.
  2. Lack of Magnitude Awareness: Models do not inherently know that 9.1 is closer to 9.2 than to 100.0.
  3. Poor Extrapolation: As noted in LIFT, models often fail to predict values outside the range of their training data because they haven't learned the underlying function, only the string patterns.

Encoding Numbers as Continuous Values

To fix this, researchers have proposed moving away from string-based representations. A landmark approach is xVal, introduced by Siavash Golkar, Miles Cranmer, and colleagues in xVal: A Continuous Number Encoding for Large Language Models.

Instead of multiple tokens for different digits, xVal uses a single dedicated [NUM] token. The actual numerical value is then used to scale the embedding of that token. This makes the model end-to-end continuous: the input is the value itself, and the model can use standard regression techniques to predict numerical outputs. This approach allows the model to capture the "smoothness" of mathematical functions, leading to better interpolation and scientific reasoning.

Other methods, like those in Vector-ICL, bypass text entirely by using lightweight projectors to map continuous vector representations (from fMRI or time-series data) directly into the LLM's internal embedding space.

Learning Proximity through Loss Functions

If we cannot change the tokenizer, we can change how the model learns. Standard LLMs use cross-entropy loss, which penalizes the model equally whether it guesses "9" when the answer is "8" or when the answer is "1,000,000."

In Regress, Don't Guess, researchers introduced Number Token Loss (NTL). NTL is a proximity-based loss function that forces the model to respect numerical distances. By penalizing the model more heavily for "far away" guesses, the model begins to organize its internal representation of numerical tokens into a coherent number line. Similarly, Laying Anchors suggests "priming" rare numbers by relating them to well-known "anchor" numbers, providing a semantic bridge to magnitude.

Applications in Tabular and Scientific Modeling

The most immediate impact of these techniques is in Large Tabular Models (LTMs). While tree-based models like XGBoost have long dominated tabular data, they lack the transfer learning capabilities of Transformers.

Models like TP-BERTa and TAPTAP attempt to bridge this by combining feature names (text) with their values (numbers). By using specialized numerical embeddings, these models can reason across different table schemas, performing tasks like missing value imputation or synthetic data generation (DP-LLMTGen) with a level of "quantitative commonsense" that standard LLMs lack.

As Charles Zhang and co-authors suggest in their survey on Multimodal Embeddings, the future of LLMs lies in moving beyond "word vectors" toward high-dimensional spaces that can natively represent the continuous nature of the physical world.

Papers in this version

Showing 60 of 342 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_number_representations_for_la_8dff03fe,
  title = {Continuous Number Representations for Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-number-representations-for-language-models/rdr_8dff03fef7494d47b7cfb295c730a8f1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_number_representations_for_la_8dff03fe_v1,
  title = {Continuous Number Representations for Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-number-representations-for-language-models/rdr_8dff03fef7494d47b7cfb295c730a8f1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}