Deep neural networks are legendary for their expressivity. Universal approximation theorems guarantee that even moderately sized networks can represent virtually any continuous function we care about. Yet, actually training these networks remains an art form plagued by rugged loss landscapes, exploding parameters, and unpredictable convergence.
Why does this gap exist? The answer lies in the mathematical bridge connecting a network's parameter space (its weights and biases) to its function space (the actual mathematical function it computes). A growing body of theoretical research reveals that this bridge is fundamentally fractured by severe mathematical instabilities.
The Geometry of Two Spaces
To build an intuition for this problem, we must separate what a network is from what a network does.
- Parameter Space ($\Theta$): The high-dimensional space of raw numerical weights. This is where gradient descent lives and travels.
- Function Space ($\mathcal{F}$): The space of the actual input-output behaviors. This is where our performance metrics (like mean squared error) live.
When we train a network, we evaluate our error in function space, but we are forced to make updates in parameter space. For optimization to be smooth and predictable, these two spaces need to be well-aligned. If two functions are nearly identical, their corresponding parameters should be close together.
Unfortunately, the mapping between these spaces is highly pathological.
[Target Function] --(Best Approximation Operator)--> [Optimal Weights]
| |
Tiny Nudge Massive Jump
v v
[New Function] ------------------------------------> [New Weights]
The Tax on Efficiency: Discontinuous Operators
The trouble begins when we look for the "best approximation operator"—the mathematical rule that takes a target function and hands us the optimal set of network weights to approximate it.
Early work in soft computing by László T. Kóczy et al. in A survey on universal approximation and its limits in soft computing techniques noted that for common architectures like radial basis functions or Heaviside perceptrons, this selection operator is inherently discontinuous.
This discontinuity is not a flaw of poor network design; rather, it is an inescapable tax on approximation efficiency. Dmitry Yarotsky proved in Optimal approximation of continuous functions by very deep ReLU networks that achieving the theoretical limits of approximation accuracy with deep, narrow ReLU networks requires a discontinuous mapping from functions to weights.
As further detailed by Ronald A. DeVore et al. in Neural network approximation, a fundamental trade-off exists: the more aggressively an architecture tries to maximize its representation efficiency, the more unstable its parameter mapping becomes. A tiny, imperceptible nudge to the target function can cause the optimal weight configuration to violently jump across the parameter space. Dmitry Yarotsky et al. later demonstrated in The phase diagram of approximation rates for deep neural networks that these highly efficient, discontinuous configurations demand extreme numerical precision, rendering them practically unreachable by standard optimization algorithms.
The Mirage of Proximity: A Lack of Inverse Stability
This phenomenon is formally characterized as a lack of inverse stability. In an inverse-stable system, proximity in the output space implies proximity in the parameter space. Neural networks completely lack this property.
In Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size, Philipp Petersen et al. proved that the parameter-to-function mapping is fundamentally non-invertible in any numerically stable way. Two networks can exhibit virtually identical input-output behavior while sitting at completely opposite ends of the parameter universe.
This lack of inverse stability directly distorts the optimization landscape. As Julius Berner et al. explain in How degenerate is the parametrization of neural networks with the ReLU activation function?, if a network mapping were inverse stable, the parameter space would inherit the nice, convex geometry of the function space. Optimization would be trivial: being close to the right function would mean being close to the right weights.
Instead, because the mapping is degenerate and unstable, the parameter landscape becomes a non-convex maze. A model can be a hair's breadth away from the perfect functional solution, yet its parameters might be trapped in a chaotic region where gradients offer no clear path forward, leading to agonizingly slow convergence.
Bypassing the Parameter Maze
Recognizing that parameter space is a minefield of discontinuities, some researchers have attempted to change the rules of optimization entirely.
In Function Space Particle Optimization for Bayesian Neural Networks, Ziyu Wang et al. proposed a method to perform particle optimization directly within the space of functions rather than the space of weights. By updating the functional outputs directly, they bypass the parameter-space bottlenecks that cause model diversity to collapse.
Yet, even function-space methods cannot fully escape the underlying pathology. To deploy a network, you must eventually translate those function-space updates back into actual network weights. The authors note that this backward step lacks rigorous theoretical guarantees. The ghost of inverse instability remains: until we find architectures with continuous parameter-to-function mappings, the optimization of deep networks will remain a battle against geometric distortion.
Key Foundational Work
Optimal approximation of continuous functions by very deep ReLU networks (2018) — contributed 3 concept elements to this direction.
A survey on universal approximation and its limits in soft computing techniques (2003) — contributed 3 concept elements to this direction.