Transformer-based models have achieved remarkable success in natural language understanding, symbolic logic, and text generation. Yet, beneath their ability to compose essays or write code lies a surprising paradox: transformers struggle with basic arithmetic and numerical reasoning. Tasks that are trivial for a simple calculator—such as multiplying two numbers, computing an average across a table, or maintaining high-precision decimals—frequently cause state-of-the-art models to fail.
To understand why this bottleneck exists, we must look at how transformers represent numbers, how they attempt to compute mathematical operations internally, and how these limitations manifest in structured data applications.
The Tokenization Trap: Numbers as Discrete Symbols
The root of the transformer's numerical deficiency begins at the very first step of pipeline processing: tokenization. Transformers process text by mapping discrete tokens into continuous vector spaces. While this works beautifully for capturing the semantic nuances of language, it introduces a fundamental flaw when dealing with numbers.
As noted by Nan Tang et al. in [Relational Pretrained Transformers towards Democratizing Data Preparation [Vision]], standard architectures represent numbers as text tokens rather than continuous magnitudes. This design choice strips numbers of their intrinsic quantitative value. A transformer does not inherently "know" that 100 is larger than 99; it merely treats them as distinct vocabulary items, much like treating "cat" and "dog" as distinct concepts.
This lack of underlying number sense manifests clearly in text embedding models. Shiyu Li et al. demonstrate in [Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings] that models frequently struggle to differentiate between distinct quantities within a query because they process numbers as discrete tokens rather than mathematical values.
Because numbers are treated as arbitrary strings, models become highly sensitive to superficial token distributions. Andrew Lan et al. show in [Automatic Short Math Answer Grading via In-context Meta-learning] that transformers often fail to differentiate between mathematically distinct operations when they share similar token distributions. If the participating numbers and symbols look alike, the model struggles to identify logical or operational errors, exposing a critical gap in its reasoning framework.
The Precision Bottleneck in Attention Layers
When a transformer attempts to solve an arithmetic problem, it must simulate the mathematical operation implicitly across its internal attention layers. Fan Zhou et al. point out in [TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data] that this implicit computation strategy is highly brittle and prone to catastrophic errors, especially when processing structured or semi-structured data.
This architectural limitation creates a hard precision ceiling. In a study on scientific computing, Jerry Weihong Liu et al. reveal in [Can Transformers Solve Least Squares to High Precision?] that standard transformers hit a strict precision bottleneck, plateauing at a $10^{-5}$ mean squared error when solving fundamental numerical problems like least squares. This failure stems directly from the model's inability to precisely implement basic linear algebra operations, such as element-wise multiplication, within standard attention mechanisms.
This reveals a profound dichotomy within the architecture. As Francois Charton argues in [Linear algebra with transformers], there is a massive performance gap between symbolic reasoning tasks—where transformers excel—and basic numerical computations like multiplication. The model can manipulate abstract tokens of logic but fails to compute their exact quantitative products.
Why Scaling and Magnitude Perception Fail to Generalize
One might assume that teaching a model to better perceive the scale of numbers would unlock arithmetic proficiency. However, Mandar Sharma et al. demonstrate in [Laying Anchors: Semantically Priming Numerals in Language Modeling] that enhancing a model's perception of numerical magnitude does not translate to proficiency in complex symbolic arithmetic. Even if a model can accurately estimate the scale of individual numbers, it still lacks the underlying logical framework required to execute multi-step mathematical operations.
Furthermore, this implicit math logic fails to generalize beyond the exact ranges seen during training. Mor Geva et al. show in [Injecting Numerical Reasoning Skills into Language Models] that even when models are injected with numerical skills via synthetic data, they fail to maintain accuracy when numerical values exceed the magnitudes or precision levels encountered during training.
This limitation also distorts relative comparisons. Tharindu Madusanka et al. find in [Not all quantifiers are equal: Probing Transformer-based language models' understanding of generalised quantifiers] that transformers struggle significantly more with proportional quantifiers (such as "most" or "few") than with absolute numerical quantifiers, showing a clear deficit in relative magnitude comparisons. Ultimately, as Jianhao Shen concludes in [Measuring Vision-Language STEM Skills of Neural Models], the persistent gap in mathematical proficiency suggests that standard scaling approaches are fundamentally insufficient for mastering formal logic and symbolic manipulation.
The Execution Gap in Tabular and Practical Applications
The real-world consequences of these arithmetic limitations are most acute when transformers interact with structured data like tables. To answer questions or generate summaries from tabular data, a model must perform multi-row mathematical operations. Wenhu Chen et al. note in [Open Question Answering over Tables and Text] that transformers frequently struggle with tasks like calculating averages or identifying extrema across multiple data points because they lack specialized multi-row logic.
This execution gap leads to severe hallucination issues during data-to-text generation. Tongliang Li et al. show in [TWT: Table with Written Text for Controlled Data-to-Text Generation] that transformers often fail to perform accurate algebraic operations, such as calculating differences or totals, even when the raw data is fully visible in the table.
The root cause is structural: standard architectures do not validate data consistency. As Zhiruo Wang et al. explain in [TUTA: Tree-based Transformers for Generally Structured Table Pre-training], table-based transformers lack the internal execution logic necessary to perform or verify mathematical operations across cells, meaning they cannot validate if the data makes numerical sense. Furthermore, without specialized embeddings for numerical order, Jonathan Herzig et al. demonstrate in [TaPas: Weakly Supervised Table Parsing via Pre-training] that transformers fail to inherently understand numerical rank or magnitude, crippling their ability to answer questions involving superlatives like "highest" or "earliest."
This arithmetic deficit prevents transformers from being safely applied to high-stakes engineering domains. For instance, Qiang Xu et al. point out in [Revolution or Hype? Seeking the Limits of Large Models in Hardware Design] that standard transformer architectures lack the high-precision numerical reasoning required for accurate timing and power analysis in electronic design, rendering them incapable of handling arithmetic-heavy physical optimization tasks without external specialized tools.
Key Foundational Work
In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering (2021) — contributed 2 concept elements to this direction.
TWT: Table with Written Text for Controlled Data-to-Text Generation (2021) — contributed 2 concept elements to this direction.
Relational Pretrained Transformers towards Democratizing Data Preparation [Vision] (2020) — contributed 2 concept elements to this direction.