Modern computing hardware is experiencing a profound architectural shift. Driven by the massive computational demands of deep learning, hardware architects have increasingly prioritized high-throughput, low-precision arithmetic units—such as 16-bit floating-point (FP16, BF16) matrix engines and 8-bit integer (INT8) accelerators.
However, this hardware evolution creates a severe challenge for scientific computing, physical simulations, and geometric optimization. These domains frequently require single-precision (FP32) or double-precision (FP64) accuracy to maintain numerical stability and prevent catastrophic cancellation. When forced to run on modern accelerators, researchers face a stark dilemma: sacrifice numerical integrity for execution speed, or abandon hardware acceleration entirely in favor of slow, CPU-bound arbitrary-precision libraries.
To resolve this tension, a prominent research direction focuses on algorithmic precision emulation. Instead of relying on native hardware support for high-precision arithmetic, these techniques cleverly orchestrate low-precision hardware operations to recover high-precision results, often surpassing the theoretical peak performance of native high-precision hardware units.
The Precision-Performance Gap
The tension between numerical accuracy and hardware throughput is not entirely new, but it has widened dramatically. Early general-purpose GPU (GPGPU) computing lacked double-precision support entirely, forcing developers to carefully recast scientific problems into graphics paradigms or accept lower fidelity (John D. Owens et al.; Paulo Cezar Pinto Carvalho et al.). Even as hardware evolved, strict bit-for-bit reproducibility requirements across architectures often penalized performance by preventing compilers from utilizing native hardware optimizations like Fused Multiply-Add (FMA) instructions (George K Thiruvathukal).
Today, the gap is driven by market economics. Specialized "Matrix Engines" (such as NVIDIA's Tensor Cores or Google's TPUs) allocate the vast majority of their silicon area to low-precision math because deep learning workloads tolerate it (Jens Domke et al.; Yi-Fan Chen et al.). Furthermore, consumer-grade graphics hardware often incorporates artificial software or hardware restrictions that throttle FP64 performance to a small fraction (e.g., 1/32 or 1/64) of the chip's total throughput (Hiroyuki Ootomo et al.).
As a result, standard optimization solvers and physical simulations running on high-throughput accelerators frequently hit a "precision floor," limiting their convergence or causing simulations to diverge entirely (Pierre Ablin et al.; Samuel S. Schoenholz).
Multi-Component Floats and Expansions
To understand how we can achieve high precision using low-precision hardware, consider how we write numbers in base-10. If you only have single-digit registers but need to store the number 13, you can split it into two registers: one storing 10 (the high-order part) and one storing 3 (the low-order part). By defining custom algebraic rules for addition and multiplication across these registers, you can perform higher-precision math using lower-precision storage.
In digital arithmetic, this is known as a Floating-Point Expansion (FPE) or a Multi-Component Float (MCF). A high-precision number $x$ is represented as the un-evaluated sum of multiple standard floating-point numbers:
$$x = x_1 + x_2 + \dots + x_n$$
where each subsequent component tracks the residual error of the previous one.
This mechanism is particularly powerful for training geometric and tree embeddings in hyperbolic space (Tao Yu et al.; Max van Spengler et al.). Hyperbolic spaces feature exponential volume growth, meaning that coordinates grow exponentially as points move away from the origin. In standard FP32 or FP16, this quickly triggers underflow, overflow, or catastrophic cancellation (the "NaN problem"). By implementing MCFs or floating-point expansions directly on the GPU, the authors bypass CPU-bound arbitrary-precision libraries, maintaining the massive parallel speed of deep learning hardware while preserving the exact geometric relationships of hierarchical data.
Compensated Matrix Math on Tensor Cores
While multi-component floats work well for element-wise operations, accelerating dense linear algebra requires mapping these components directly onto hardware matrix multiplication units.
To perform a high-precision matrix multiplication $C = A \times B$ using low-precision hardware, the input matrices are split into high-order and low-order matrices based on their bit representations:
$$A = A_{\text{high}} + A_{\text{low}}$$ $$B = B_{\text{high}} + B_{\text{low}}$$
The full multiplication can then be expanded into a sum of cross-products:
$$C = (A_{\text{high}} \times B_{\text{high}}) + (A_{\text{high}} \times B_{\text{low}}) + (A_{\text{low}} \times B_{\text{high}}) + (A_{\text{low}} \times B_{\text{low}})$$
Each of these sub-multiplications can be executed natively at lightning speed on low-precision hardware units, and the results are subsequently accumulated into a high-precision container.
High-Precision Matrix A High-Precision Matrix B
[ A_high | A_low ] [ B_high | B_low ]
\ / \ /
\ / \ /
X <-- Four Low-Precision Cross-Multiplications --
/ \
/ \
(A_high * B_high) + (A_high * B_low) + (A_low * B_high) + (A_low * B_low)
========================================================================
High-Precision Result Matrix C
Hiroyuki Ootomo et al. demonstrated that by carefully managing internal hardware rounding quirks during this splitting process, they could recover exact FP32 accuracy from FP16 Tensor Cores. Remarkably, because Tensor Cores are so fast, executing these multiple sub-multiplications and combining them still runs up to 2.6× faster than the hardware's native, un-accelerated FP32 theoretical peak. The authors later extended this paradigm to execute double-precision matrix multiplication (DGEMM) on integer matrix units (IMUs), effectively transforming consumer-grade AI engines into high-performance scientific computing tools (Hiroyuki Ootomo et al.).
Mixed-Precision Refinement and Algorithmic Reformulation
A complementary approach operates at a higher algorithmic level, combining low-precision approximations with high-precision corrections, or reformulating the underlying mathematics to be inherently more resilient to low precision.
Iterative Refinement
Instead of emulating precision at every single clock cycle, mixed-precision iterative refinement uses low precision for the most computationally expensive parts of an algorithm and high precision only where errors accumulate. For instance, when solving a linear system $Ax = b$:
- Compute an approximate solution $x_0$ rapidly using low-precision hardware.
- Compute the residual error $r = b - Ax_0$ using high precision to avoid cancellation errors.
- Solve a correction equation $Ad = r$ using low precision.
- Update the solution $x_1 = x_0 + d$.
This loop repeats until convergence. Because the bulk of the work (factoring or multiplying matrices) happens in low precision, the algorithm runs at accelerator speeds, while the high-precision residual step guarantees double-precision accuracy upon convergence (Peter Benner et al.; Sergio Barrachina 0001 et al.).
Algorithmic Reformulation
Sometimes, the requirement for high precision is an artifact of how a problem is mathematically framed rather than the problem itself. A classic example occurs in large-scale 3D reconstruction, known as bundle adjustment. Traditional solvers construct a Hessian matrix ($J^T J$), an operation that squares the condition number of the problem and mandates FP64 arithmetic to prevent numerical collapse.
Nikolaus Demmel bypassed this requirement by introducing Square Root Bundle Adjustment ($\sqrt{\text{BA}}$). Instead of forming the Hessian, the method performs a QR decomposition directly on the Jacobian matrix $J$. Because it avoids squaring the condition number, the formulation is inherently stable enough to run entirely in single precision (FP32) while matching the accuracy of standard double-precision solvers—effectively doubling the computational speed and memory efficiency on standard hardware accelerators.
Key Foundational Work
A Survey of General-Purpose Computation on Graphics Hardware (2005) — contributed 2 concept elements to this direction.
Java Grande Forum Report: Making Java Work for High-End Computing (1998) — contributed 2 concept elements to this direction.