For decades, the theoretical foundation of deep learning has rested on "Universal Approximation Theorems." These theorems provide a comforting mathematical safety net: they prove that for almost any complex function, there exists a neural network configuration capable of approximating it with arbitrary precision. However, a growing body of research suggests that this "existence" is a theoretical mirage that masks a fundamental crisis in practical AI.
The core problem is the Reachability Gap: the fact that proving a solution exists in the weight space is not the same as proving that an algorithm like Stochastic Gradient Descent (SGD) can ever find it.
The Existence Trap
In classical approximation theory, researchers like Dmytro Perekrestenko and Helmut Bölcskei have shown that deep ReLU networks are incredibly efficient. In The universal approximation power of finite-width deep ReLU networks, they demonstrate that even narrow networks can approximate oscillatory signals and fractals with error rates that drop exponentially with depth.
But as Anastasis Kratsios and Daniel M. Roy argue in Beyond Universal Approximation Theorems, these proofs are often "constructive"—they provide a manual recipe for setting weights to achieve a goal. There is no mathematical guarantee that the path taken by gradient-based optimization will ever intersect with these hand-crafted, optimal structures. We are essentially proving that a needle exists in a haystack without proving that our magnet is strong enough to pull it out.
The Digital Blind Spot and Uncomputability
The gap isn't just a matter of "hard" optimization; it may be a matter of fundamental uncomputability. Holger Boche and Gitta Kutyniok identify what they call a "digital blind spot" in Computability of Optimizers. They prove that for many common AI problems, the optimal value might be easy to calculate, but the optimizer—the specific set of weights that achieves that value—cannot be computed by any digital hardware (Turing machine), regardless of processing power.
This suggests that our reliance on GPUs and standard numerical floating-point math creates an inherent ceiling. We might be searching for optimal parameters that our hardware is mathematically barred from representing or reaching.
The Stability Paradox
This reachability crisis is most visible in the struggle for AI robustness. In The mathematics of adversarial attacks in AI, Alexander Bastounis and Anders C. Hansen provide a sobering proof: while stable and accurate neural networks are mathematically proven to exist, no standard randomized algorithm can reliably compute them.
This explains why deep learning remains unstable despite the existence of stable solutions. As David Liu and Hansen explore in Do stable neural networks exist for classification problems?, we are caught in a paradox where the models we can train are destined to be fragile, while the robust models we want remain mathematically "out of reach" for current training regimes.
Bridging the Gap via Algorithmic Approximation
To solve this, the field is shifting from "Existence Theory" to "Algorithmic Approximation Theory." Instead of asking "What can a network represent?", researchers are asking "What can a network learn?"
Several directions are emerging to bridge this divide:
- Constructive Learning: Shaobo Lin proposes moving away from iterative backpropagation entirely. In Constructive neural network learning, Lin suggests "building" networks directly from the geometry of the data (using Voronoi partitions), which bypasses the uncertainties of the loss landscape.
- Adaptive Basis Viewpoints: Researchers like Eric C Cyr suggest that the disconnect is a failure of our optimizers. In Robust Training and Initialization of Deep Neural Networks, they propose hybrid optimizers that treat the network as an adaptive basis set, combining least squares with gradient descent to force the model toward theoretically sound solutions.
- Regularity-Preserving Networks: Kratsios and Ruiyang Hong have developed architectures that are "universally" well-behaved. In Bridging the Gap Between Approximation and Learning, they use geometric Kuhn triangulation to ensure that the network maintains the smoothness of the target function, making the optimal configuration more "discoverable" by standard dynamics.
The consensus across these 118 papers is clear: the next frontier of AI theory isn't proving that neural networks are powerful, but understanding why—and when—our algorithms are actually capable of wielding that power.