Nonclosedness of sets of neural networks in Sobolev spaces

Scott Mahan, Emily J. King, Alex CloningerNeural Networks 2021 · OpenReview

TL;DR

This paper proves that sets of neural networks with a fixed architecture are generally not closed in Sobolev spaces. This means neural networks can approximate functions that are not themselves realizable by a network, even with smooth activation functions. The research also shows that this approximation often requires network parameters (weights) to grow without bound, and it characterizes this relationship between approximation error and parameter growth.

What Problem Does This Paper Address?

In approximation theory, a key question is whether a set of functions (like those representable by neural networks) is "closed" under a given norm. If a set is closed, then any sequence of functions within that set that converges to some limit must have that limit also within the set. Conversely, if a set is not closed, it implies that functions outside the set can be approximated arbitrarily well by functions within the set.

Previous work has shown that neural network function sets are not closed in simpler function spaces like $L^p$ or $C^0$. However, the behavior in Sobolev spaces, which incorporate derivatives and are thus stronger measures of smoothness, was less understood. The authors investigate this, specifically for neural networks with a fixed architecture and a fixed activation function. This is crucial because in practice, network architectures are often fixed during training. Understanding closedness in Sobolev spaces is important for tasks like Sobolev training, where derivatives of the target function are used, and for analyzing the theoretical expressivity of neural networks.

Methods

The core of the paper's theoretical contribution lies in constructing explicit sequences of neural networks. For an activation function $\rho$ that is $m$-times differentiable but not $(m+1)$-times differentiable, they construct a sequence of networks whose realizations converge in the $(m-1)$-th order Sobolev norm to a function that is not a neural network. This directly demonstrates non-closedness. They extend this to the $m$-th order Sobolev norm under stronger conditions on $\rho$'s $m$-th derivative. For real analytic activation functions, they show non-closedness in any order Sobolev space.

To analyze parameter growth, they consider a specific sequence of networks approximating the derivative of the activation function. They then derive a relationship showing that the $L^p$ approximation error decreases approximately inversely proportional to the network's total norm (a measure of its weights and biases).

Experimentally, they use "Sobolev training" with an Adam optimizer to train networks to approximate target functions in Sobolev norm. They investigate this for various activation functions (ELU, sigmoid, ISRLU) and target functions that are known not to be realizable by networks of the same architecture.

Datasets

Not discussed in the paper. The experiments involve generating synthetic target functions rather than using pre-existing datasets.

Key Insights and Ideas

  • Neural networks are "thin" in Sobolev spaces: Even with smooth activation functions, the set of functions a fixed-architecture neural network can represent is not "complete" in Sobolev spaces. This means there are smooth functions that networks can approximate arbitrarily well, but cannot perfectly represent.
  • Smoothness matters, but doesn't guarantee closedness: The degree of differentiability of the activation function dictates the order of the Sobolev space in which non-closedness is proven. More differentiable activation functions lead to non-closedness in higher-order Sobolev spaces.
  • Analytic activation functions are "very thin": For real analytic activation functions, the set of realizable networks is not closed in any order Sobolev space, highlighting their broad approximation capabilities.
  • Approximation implies unbounded parameters: When a neural network approximates a non-network target function in Sobolev norm, it necessitates that the network's parameters (weights and biases) grow without bound. This provides a theoretical basis for the phenomenon of exploding weights observed during training.
  • Parameter growth is predictable (in some cases): For specific scenarios (e.g., approximating the derivative of the activation function), the rate of parameter growth is inversely related to the approximation error. This suggests a trade-off between accuracy and model complexity.

Key Results

  • Theorem 3.1: For an $m$-times differentiable activation function $\rho$ (but not $(m+1)$-times), sets of realized neural networks are not closed in $W^{m-1,p}$ for $p \in [1, \infty]$. They are also not closed in $W^{m,p}$ under slightly stronger conditions on $\rho^{(m)}$.
  • Theorem 3.3: For real analytic, bounded, non-constant activation functions, sets of realized neural networks are not closed in $W^{k,p}$ for any order $k \in \mathbb{N}$.
  • Proposition 3.6: For certain activation functions (softsign, ELU, ISRLU), the $L^p$ approximation error when learning the derivative of the activation function is approximately inversely proportional to the network's total norm ($|| \Phi ||_{total}$).
  • Experimental Results (Figures 1, 2, 4): Simulations with ELU, sigmoid, and ISRLU activation functions demonstrate that networks can indeed approximate non-network target functions in Sobolev norm, with observed parameter growth that aligns with the theoretical predictions. The $L^2$ and Sobolev errors decrease rapidly, while network norms increase steadily.

Limitations and Future Work

  • Generalization of Parameter Growth: While Proposition 3.6 provides a characterization for a specific case (learning the derivative), the paper suggests that describing the relationship between error and parameter growth for arbitrary target functions, network architectures, and higher-order Sobolev norms is an interesting area for future research.
  • Practical Implications of Unbounded Parameters: The theoretical finding of unbounded parameters implies that training might be difficult in practice, or that regularization techniques are essential to prevent this. The paper notes that while Sobolev training allows approximation of non-network functions, it doesn't prevent parameter growth.
  • Deep Networks: The experiments were conducted with shallow networks. While the authors expect similar non-closedness phenomena for deep networks due to their increased expressivity, this was not directly tested.
  • Specific Architectures: The theoretical results often focus on specific network structures (e.g., $L-1$ layers with $N_i=1$ for $i<L-1$, and $N_L=1$). Extending these to more general architectures could be a future direction.

Summary updated

Abstract

We examine the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces. For an exactly m<math><mi is="true">m</mi></math>-times differentiable activation function ρ<math><mi is="true">ρ</mi></math>, we construct a sequence of neural networks (Φn)n∈N<math><msub is="true"><mrow is="true"><mrow is="true"><mo is="true">(</mo><msub is="true"><mrow is="true"><mi is="true">Φ</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><mo is="true">)</mo></mrow></mrow><mrow is="true"><mi is="true">n</mi><mo is="true">∈</mo><mi mathvariant="double-struck" is="true">N</mi></mrow></msub></math> whose realizations converge in order-(m−1)<math><mrow is="true"><mo is="true">(</mo><mi is="true">m</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">−</mo><mn is="true">1</mn><mo is="true">)</mo></mrow></math> Sobolev norm to a function that cannot be realized exactly by a neural network. Thus, sets of realized neural networks are not closed in order-(m−1)<math><mrow is="true"><mo is="true">(</mo><mi is="true">m</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">−</mo><mn is="true">1</mn><mo is="true">)</mo></mrow></math> Sobolev spaces Wm−1,p<math><msup is="true"><mrow is="true"><mi is="true">W</mi></mrow><mrow is="true"><mi is="true">m</mi><mo is="true">−</mo><mn is="true">1</mn><mo is="true">,</mo><mi is="true">p</mi></mrow></msup></math> for p∈[1,∞)<math><mrow is="true"><mi is="true">p</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mrow is="true"><mo is="true">[</mo><mn is="true">1</mn><mo is="true">,</mo><mi is="true">∞</mi><mo is="true">)</mo></mrow></mrow></math>. We further show that these sets are not closed in Wm,p<math><msup is="true"><mrow is="true"><mi is="true">W</mi></mrow><mrow is="true"><mi is="true">m</mi><mo is="true">,</mo><mi is="true">p</mi></mrow></msup></math> under slightly stronger conditions on the m<math><mi is="true">m</mi></math>th derivative of ρ<math><mi is="true">ρ</mi></math>. For a real analytic activation function, we show that sets of realized neural networks are not closed in Wk,p<math><msup is="true"><mrow is="true"><mi is="true">W</mi></mrow><mrow is="true"><mi is="true">k</mi><mo is="true">,</mo><mi is="true">p</mi></mrow></msup></math> for any k∈N<math><mrow is="true"><mi is="true">k</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mi mathvariant="double-struck" is="true">N</mi></mrow></math>. The nonclosedness allows for approximation of non-network target functions with unbounded parameter growth. We partially characterize the rate of parameter growth for most activation functions by showing that a specific sequence of realized neural networks can approximate the activation function’s derivative with weights increasing inversely proportional to the Lp<math><msup is="true"><mrow is="true"><mi is="true">L</mi></mrow><mrow is="true"><mi is="true">p</mi></mrow></msup></math> approximation error. Finally, we present experimental results showing that networks are capable of closely approximating non-network target functions with increasing parameters via training.
BibTeX
@inproceedings{mahan2021nonclosedness,
  author = {Scott Mahan and Emily J. King and Alex Cloninger},
  title = {{Nonclosedness of sets of neural networks in Sobolev spaces}},
  booktitle = {Neural Networks},
  year = {2021},
  url = {https://openreview.net/forum?id=yumpbbpYHe},
}
Figure