Abstract
Significant inaccuracy often occurs during the process of mathematical calculation due to the digit limitation of floating point, which may lead to catastrophic loss. Normally, people believe that adjustment of floating-point precision is an effective way to solve this problem, since high-precision floating-point has more digits to store information. Thus, it is a prevalent method to reduce the inaccuracy in much floating-point related research, that performing all the operations with higher precision. However, we discover that some operations may lead to larger error in higher precision. In this paper, we define this kind of operation that generates large error due to precision adjustment a precision-specific operation. Furthermore, we propose a light-weight searching algorithm for detecting precision-specific operations and figure out an automatic processing method to fixing them. In addition, we conducted an experiment on the scientific mathematical library of GLIBC. The result shows that there are many precision-specific operations, and our fixing approach can significantly reduce the inaccuracy.
Introduction: The Paradox of Precision
In the world of scientific computing, floating-point numbers are the standard approximations for real numbers. However, because they rely on a finite number of bits, they are inherently prone to rounding errors, accumulation issues, and catastrophic cancellation. For decades, the industry's "common sense" solution has been straightforward: if a calculation isn't accurate enough, use more bits. Moving from single precision (32-bit) to double precision (64-bit) or higher is the default strategy for mitigating numerical instability.
As shown in the figure above, even widely used tools like Matlab and the Windows calculator can produce wildly different—and often incorrect—results for complex operations when compared to a reliable, high-precision reference. While increasing precision usually helps, this paper identifies a counter-intuitive phenomenon: "precision-specific operations," where increasing the number of bits actually decreases accuracy.
The Problem: Precision-Specific Operations
The authors discovered that some mathematical algorithms are meticulously hand-tuned by developers to exploit the specific bit-layout of a certain precision level (typically IEEE 754 double precision). When these algorithms are executed in a high-precision environment—such as a 120-bit "shadow" execution used for debugging—the underlying "tricks" break. This leads to relative errors that can be orders of magnitude worse than those found in the original, lower-precision code.
The Union Trap and Bit-Level Manipulation
One primary cause of this paradox is the "Union Trap." Developers often use C unions to treat a floating-point number as an integer array, allowing them to modify specific bits directly for performance or algorithmic reasons.
As illustrated in Figure 1, different precision formats have entirely different bit-layouts. A 32-bit float allocates 8 bits to the exponent and 23 to the fraction, while a 64-bit double uses 11 and 52 bits, respectively. If a developer writes code to flip the 31st bit to change a sign in single precision, that same logic becomes nonsensical when the precision is adjusted to a format where the sign bit is at index 63 or 127.
The "Rounding Trick"
Another common pattern involves using large constants to force rounding. For instance, adding a constant like $1.5 \times 2^{52}$ to a number can force the CPU to shift the mantissa, effectively rounding the value to an integer. In a high-precision environment with a much longer mantissa, the shift does not "push out" the fractional bits as intended, causing the "trick" to fail and introducing significant error.
Methodology: Detection and Automatic Processing
To address these paradoxical errors, the researchers developed a two-step framework:
- Detection via FPdebug: The authors utilized
FPdebug, a tool that uses binary translation to execute code in two modes simultaneously: the original precision and a "shadow" higher precision. By monitoring thousands of inputs, a statistical searching algorithm identifies "stable" operations where the high-precision version consistently and significantly deviates from the expected mathematical result. - Automatic Processing (The Fix): Once a precision-specific operation is identified, the tool inserts a "mixed-precision" wrapper:
reducePrec: Immediately before the problematic instruction, the high-precision "shadow" value is rounded back down to the original precision.resumePrec: After the instruction executes in its intended original precision, the result is converted back up to high precision to maintain the benefits of higher bits for the rest of the program.
Key Results and Evaluation
The authors evaluated their approach on the GNU C Library (GLIBC) version 2.19, specifically focusing on standard math.h functions like acos, exp, and sin. Using a reliable multi-precision library as the "ground truth," they found:
- Widespread Issues: Out of 54 tested functions in GLIBC, 111 precision-specific operations were detected.
- Significant Accuracy Gains: In the case of the
expfunction, the standard high-precision version actually performed worse than the original. However, after applying the "Mixed Precision" (MP) fix, 100% of the inputs achieved higher accuracy than both the original and the un-fixed high-precision versions. - Broad Success: For nearly all tested functions, the fixing method improved performance for over 90% of input values compared to standard high-precision execution.
Conclusion
This research highlights that precision is a tool, not a guarantee of accuracy. Software is often deeply "tuned" to its hardware representation. Blindly upgrading precision without considering algorithmic intent can be counterproductive. By identifying and strategically downscaling precision for specific operations, developers can harness the benefits of high-precision computing without falling into the traps of hardware-dependent optimizations.
Summary updated