In the world of deep learning theory, we often take for granted that if we keep training a neural network of a fixed size, we will eventually converge to a "best" version of that network. But mathematically, this assumes that the set of all functions representable by a specific architecture is closed—meaning that if a sequence of networks converges to a limit, that limit is also a network of the same size.
A recent paper by Eloi Martinet and Leon Bungert, "On the (non-)closedness of the set of ReLU networks," dives deep into this topological rabbit hole. They discover a striking divide: while shallow ReLU networks are remarkably well-behaved, adding just one more hidden layer breaks the fundamental property of closedness.
TL;DR
The authors prove that shallow (one-hidden-layer) ReLU networks form a "weakly closed" set in Sobolev spaces, guaranteeing that optimal networks exist for many common training objectives. However, they disprove a major conjecture by showing that networks with two or more hidden layers are not closed, meaning their parameters can blow up to infinity as they attempt to reach a limit that the architecture itself cannot represent.
What Problem Does This Paper Address?
When we train a neural network to minimize a loss function (like mean squared error or a physics-informed energy), we are essentially searching through a "realization set"—the collection of all functions our chosen architecture can produce.
For this search to be mathematically sound, we need to know if a "best" network actually exists within that set. If the set isn't closed, a sequence of networks could get better and better, but their limit might be a function that requires a larger or different architecture to represent. This is known as a "coercivity gap," where network parameters (weights and biases) escape to infinity even as the function they represent seems to converge.
Previous research had shown that for most smooth activation functions, these sets are not closed. For ReLU, it was known that shallow networks are closed in simple uniform topologies, but the status of deeper networks and more complex function spaces (like Sobolev spaces used in scientific machine learning) remained a mystery.
Methods
The authors employ rigorous functional analysis, focusing on Sobolev spaces ($W^{1,p}$), which measure not just function values but also their derivatives. This is particularly relevant for methods like Deep Ritz or Sobolev training, where gradients matter.
The Shallow Case: Sorting and Collapsing
To prove closedness for shallow networks, the authors developed a "sorting" strategy for neurons. As a sequence of networks converges, they track the "break hyperplanes" (where the ReLU function flips from zero to linear).
- They show that neurons whose hyperplanes leave the domain effectively become simple affine functions.
- For neurons whose hyperplanes stay inside, they prove that clusters of neurons sharing a limiting hyperplane must "collapse" into a single ReLU neuron in the limit.
The Deep Case: A Constructive Counterexample
To prove non-closedness for deeper networks, the authors explicitly constructed a sequence of networks with two hidden layers (widths 4 and 6). They showed this sequence converges to a piecewise affine function that cannot be represented by any network of that same size. This construction relies on the fact that the first layer can "mimic" an identity mapping with high precision, allowing the second layer to create complexities that "break" the closedness property in the limit.
Key Insights and Ideas
- Shallow Rigidity: Shallow ReLU networks possess a geometric rigidity in Sobolev spaces. The requirement for derivative convergence prevents the "jump" discontinuities that typically cause non-closedness in simpler $L^p$ spaces.
- The Danger of Depth: Depth provides enough functional flexibility to "hide" complexity. A sequence of deep networks can converge to a function that "looks" like it belongs to the architecture but actually requires more neurons to represent exactly.
- Disproving a Conjecture: The paper disproves a conjecture by Petersen et al. (2021), which suggested that ReLU networks of fixed complexity should be closed in the space of continuous functions ($C(\Omega)$). The authors show that without explicit parameter bounds, this is false for deep networks.
- Theoretical Justification for Weight Decay: Because deep architectures are not closed, training can lead to diverging parameters. This provides a strong theoretical argument for weight decay (L2 regularization), which forces parameters to stay bounded and ensures the existence of a minimizer.
Key Results
- Theorem 2.3 (Closedness): The set of shallow ReLU networks with at most $N$ neurons is weakly closed in $W^{1,p}(\Omega)$. This ensures that variational problems like the Deep Ritz method always have a solution when using shallow networks.
- Theorem 3.1 (Non-closedness): For dimensions $d \ge 2$, the set of networks with two hidden layers is not closed in either continuous or Sobolev spaces.
- Minimal-Norm Selection: The authors prove that as the weight decay penalty goes to zero, the resulting minimizers converge to the network with the smallest possible parameter norm that still minimizes the original loss.
Limitations and Future Work
- The 1D Gap: The counterexample for deep networks requires at least two dimensions. Whether deep ReLU networks are closed in 1D remains an open question.
- Optimization Dynamics: While the paper proves the existence (or lack thereof) of minimizers, it doesn't address whether standard algorithms like Stochastic Gradient Descent (SGD) can actually find them.
- Other Activations: The results are specific to ReLU. Exploring how "leaky" ReLUs or other variants affect these topological properties is a natural next step.
In summary, this work highlights that "deeper is better" comes with a mathematical price: a loss of topological stability that necessitates careful regularization to keep our models well-behaved.
Summary updated