Introduction
Formal verification is often considered the "gold standard" for AI safety, aiming to provide mathematical proofs that a neural network will behave correctly under all possible inputs. However, this research exposes a critical "implementation gap": verifiers that are mathematically proven to be sound for theoretical models often fail to provide safety guarantees for networks running on real-world hardware. By exploiting the non-associative nature of floating-point arithmetic, the authors demonstrate that "deployment-sensitive backdoors" can bypass every state-of-the-art verifier tested.
The Problem: Theoretical vs. Practical Soundness
Most formal verifiers operate on a theoretical model, treating the neural network as a sequence of real-valued mathematical operations. In reality, networks are deployed using floating-point arithmetic. Unlike real numbers, floating-point operations are non-associative: $(a + b) + c$ does not always equal $a + (b + c)$.
Deployment environments—varying by GPU architecture, software libraries (such as PyTorch vs. Flux), or parallelization settings—can change the order of these operations or the precision used. The authors argue that theoretical soundness (bounding the ideal model) does not imply practical soundness (bounding the actual deployed code). This discrepancy creates a vulnerability where a network might pass a safety check but behave maliciously once deployed in a specific environment.
Methodology
The authors combined formal proofs with empirical "red-teaming" to demonstrate this gap through three primary steps:
- Formal Analysis of Soundness: They proved that standard verification techniques—including Interval Bound Propagation (IBP), Polyhedra-based methods (DeepPoly), and Zonotope-based methods (DeepZ)—are not practically sound. If a verifier does not account for the exact order of operations used in deployment, it can fail to bound the output correctly.
- Deployment-Sensitive Backdoors: The authors designed "detector neurons" that act as environment sensors. These neurons are engineered to output zero (triggering normal behavior) in one environment but a non-zero value (triggering a backdoor) in another.
- Precision Detectors: Triggered by the difference between 32-bit and 64-bit floating-point representations.
- Expression Tree Detectors: Triggered by specific summation orders (e.g., adding numbers in a decreasing order vs. the default order).
- Adversarial Network Construction: These detectors were inserted into a standard MNIST classifier. The resulting backdoored network behaves perfectly on the verifier's machine but flips its predictions entirely when deployed in the targeted environment.
Key Insights
The Implementation Gap as a Security Risk
The implementation gap is not merely a minor numerical error; it represents a fundamental difference in mathematical objects. This allows an attacker to hide a "logic bomb" in a network that only activates on a specific hardware or software stack.
Verification is Environment-Blind
Current verifiers treat the deployment environment as an obstacle to be ignored or handled with simple rounding bounds. The authors argue that the environment must become a fundamental part of the verification effort.
The Complexity of Summation
Even simple summation is difficult to verify. Because there are $n!$ possible ways to order a sum of $n$ numbers, finding the specific order that produces the minimum or maximum floating-point result is likely an NP-hard problem. Furthermore, because parallelized environments often sum values in a non-deterministic order, a verifier cannot easily prove a property holds unless it accounts for all possible execution orders.
Results and Universal Failure
The authors tested their backdoored networks against a pool of top-tier verifiers, including MIPVerify, DeepPoly, DeepZ, β-CROWN, and MN-BaB.
- Universal Failure: Every single verifier was fooled by the Precision attack and the most difficult Expression Tree attack (Order3).
- False Sense of Security: In many cases, verifiers reported 97% robust accuracy on networks that actually had 0% accuracy in the targeted deployment environment.
- Accidental Success: While some verifiers caught simpler attacks (Order1 and Order2), the authors show this was often due to "lucky accidents" where the verifier's internal math happened to overlap with the backdoor's trigger, rather than true practical soundness.
Limitations and Future Directions
The theoretical proofs in this study focused primarily on summation. While neural networks are largely composed of sums (dot products and convolutions), future work could extend this analysis to other operations like activations or transcendental functions.
Achieving true practical soundness—covering all possible execution orders—may be computationally prohibitive. Consequently, the authors call for the community to move toward environment-aware verification, where the specific hardware and software specifications are treated as necessary inputs to the verification algorithm.
Summary updated