How Finite Precision Constraints Reshape Neural Network Expressivity

295 concepts160 papers2 proposals

Theoretical deep learning often operates in an idealized mathematical paradise. In this realm, neural networks are built on the continuum of real numbers ($\mathbb{R}$), possessing infinite numerical precision. Under these assumptions, classic proofs demonstrate that Recurrent Neural Networks (RNNs) and Transformers are Turing-complete, capable of simulating any algorithm or representing any continuous function.

However, physical digital hardware—GPUs, TPUs, and neuromorphic chips—does not live in a continuum. It operates on a discrete grid of finite-precision representations, such as IEEE 754 float32, bfloat16, or quantized integers.

This discrepancy is not just a minor implementation detail; it fundamentally alters what neural networks can compute. When we constrain a model to finite precision, its theoretical capabilities can collapse, shifting it into entirely different computational complexity classes. Conversely, understanding these constraints reveals why practical techniques like Chain-of-Thought prompting or pause tokens are mathematically necessary.

The Collapse of Infinite Information

To understand why precision matters, consider how classic proofs establish the Turing completeness of architectures like RNNs or Transformers. These proofs frequently rely on a trick: encoding an unbounded amount of information or an infinite memory stack within the infinite decimal places of a single real-valued activation.

As Pérez Pérez et al. demonstrated, restricting activations to fixed-precision formats significantly reduces this computational capacity, preventing models from achieving true Turing completeness. When you restrict a network to a fixed number of bits, you cap the total number of reachable internal states. As a result, an architecture that is theoretically omnipotent over real numbers degrades into a finite automaton or a shallow circuit when deployed on actual hardware.

This gap between continuous theory and discrete reality manifests across several domains:

  • Inverse Problems: Boche et al. proved that while neural networks can mathematically solve complex inverse problems (like MRI reconstructions) in a continuum, those solutions are often algorithmically unreachable or unrepresentable when executed on digital hardware restricted to computable numbers.
  • Language Modeling: Nowak et al. showed that while recurrent language models can theoretically represent any computable probability distribution using exact rational arithmetic, finite-precision floating-point numbers strictly limit the reachable state space, invalidating standard asymptotic bounds.
  • Graph Drawing: In classical algorithms, Eades et al. noted that assuming a real RAM model creates a severe gap in practice; certain geometric layouts require exponential area, meaning standard floating-point representations lack the precision to preserve basic topological structures.

The Counting Ceiling and Floating-Point Absorption

To build intuition for how finite precision breaks down a model's logic, we can look at the mechanics of floating-point arithmetic—specifically, the phenomenon of numerical absorption.

In an idealized system, a network can count to infinity by repeatedly adding $1$ to an accumulator state: $s_{t+1} = s_t + 1$. But in floating-point math, numbers are represented with a significand and an exponent. As the accumulator $s_t$ grows large, its exponent increases, causing the gap between consecutive representable values to widen. Eventually, the gap becomes greater than $1$. At that exact point, evaluating $s_t + 1$ rounds right back down to $s_t$. The addition of a small value to a sufficiently large value results in absolutely no change.

Idealized Real Numbers:
0 -> 1 -> 2 -> ... -> 16,777,216 -> 16,777,217 -> 16,777,218

FP32 Precision (Significand limits):
0 -> 1 -> 2 -> ... -> 16,777,216 -> 16,777,216 -> 16,777,216  (Counting stalls!)

Ahvonen formalized this exact mechanism in the context of Graph Neural Networks (GNNs). While real-valued GNNs can count neighbors or aggregate graph elements without bound, floating-point GNNs hit a hard "counting ceiling." Because of non-associativity and rounding errors inherent to formats like bfloat16, the model becomes physically incapable of distinguishing graph properties beyond a certain scale.

Why Chain of Thought and Pause Tokens are Mathematically Necessary

If finite precision cripples a network's ability to maintain deep internal state, how do modern models solve complex, multi-step reasoning problems? The answer lies in shifting the computational burden from numerical depth to sequence length.

Li et al. analyzed this through the lens of circuit complexity. They showed that under constant-precision modeling (e.g., FP16 or BF16), fixed-depth Transformers are confined to lower complexity classes, such as shallow boolean circuits. Without generating intermediate tokens, it is mathematically impossible for these models to solve inherently serial problems like counting or parity checking because they cannot store the intermediate state internally.

This provides a rigorous explanation for why Chain-of-Thought (CoT) prompting works. By forcing the model to write out its reasoning step-by-step, the physical context window acts as an external, discrete memory bank.

Similarly, London et al. proved that adding "pause tokens" (filler symbols like "...") strictly increases the expressivity of constant-depth, fixed-precision Transformers. These tokens grant the model extra computational steps to manipulate its internal representations before committing to an output, effectively expanding its operational workspace without altering the underlying hardware precision.

The One-Bit Threshold and Bounded Universality

Given these severe constraints, one might wonder if finite-precision networks lose all their theoretical guarantees. Fortunately, recent work shows that the transition from infinite to finite precision behaves predictably, characterized by sharp thresholds and bounded approximations.

The Precision Cliff

Expressivity does not degrade smoothly; it frequently hits performance cliffs. Chakrabarti et al. demonstrated a "one-bit threshold" for Transformer expressivity. They proved that for any precision level $p$, there exist fundamental logical primitives—such as exact string matching or equality checking—that a single-layer Transformer can execute perfectly with $p$ bits, but becomes mathematically incapable of solving if the bit-depth is reduced to $p-1$. This sensitivity highlights why low-bit quantization (e.g., Int4) often breaks down on precise reasoning tasks.

Bounded Universality

Despite these thresholds, finite-precision networks still retain a form of hardware-constrained universality. Park et al. and Hwang et al. proved that neural networks operating under standard floating-point arithmetic (accounting for non-associative rounding and overflows) can exactly represent almost any function that maps floating-point inputs to floating-point outputs.

The critical caveat is that their approximation guarantees are bounded by an intrinsic error threshold defined by the distance to the nearest representable floating-point value. Zero-error approximation of arbitrary real numbers is physically impossible, but optimal approximation up to the hardware's resolution limit is entirely achievable.

Conclusion

When evaluating what a neural network can compute, we cannot separate the architecture from the arithmetic format. Finite precision constraints dismantle the assumption of infinite internal state, imposing hard ceilings on counting, logic, and memory. Yet, architectures adapt: the success of modern sequence models relies on leveraging token generation to bypass these hardware boundaries, turning a static numerical bottleneck into a dynamic, distributed computation.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 169 papers
  2. Version 1 · 2026-10-04 · 266 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_numerical_precision_constraints_0695e3e5,
  title = {numerical precision constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-precision-constraints/rdr_0695e3e5a0e64c70a60feee080a59a96},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_numerical_precision_constraints_0695e3e5_v2,
  title = {numerical precision constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-precision-constraints/rdr_0695e3e5a0e64c70a60feee080a59a96/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake