Abstract
Traditional neural networks rely on a discrete sum of hidden units. This paper explores the theoretical and practical implications of extending these architectures to an uncountable (infinite) number of hidden units by replacing the summation with an integral. We introduce two primary implementations: Affine Neural Networks, which utilize a piecewise affine weight function to achieve superior approximation bounds and reduced gradient saturation, and a non-parametric kernel machine. The latter leads to the derivation of a "sign kernel" that is remarkably effective and requires no hyperparameter tuning, offering a robust alternative to standard Gaussian kernels.
Introduction
In the 1990s, Radford Neal demonstrated that neural networks with an infinite number of hidden units and Gaussian priors on weights converge to Gaussian processes. While this provided a theoretical bridge between deep learning and kernel methods, practical implementation remained challenging. Standard architectures still utilize a finite, discrete set of $h$ hidden units.
This work addresses the practical implementation and training of networks with a continuous, uncountable number of hidden units. By shifting the mathematical foundation from discrete sums to integrals, the authors aim to leverage the performance advantages of infinite-width limits while maintaining computational feasibility.
Methodology
The fundamental shift in this approach is the replacement of the standard output equation: $$f(x) = \sum_{i=1}^h a_i g(x \cdot V_i)$$ with an integral over a continuous index $u$: $$f(x) = \int a(u) g(x \cdot V(u)) du$$
To make this formulation practical, two distinct strategies are proposed:
1. Affine Neural Networks
Rather than integrating over an arbitrary space, the weight function $V(u)$ is restricted to a finite-dimensional representation. While standard networks can be seen as using a piecewise constant approximation of this integral, Affine Neural Networks use a piecewise affine (continuous and linear by parts) function. This is analogous to moving from the rectangle method to the trapezoid method in numerical integration.
2. Non-Parametric Kernel Machine
By integrating over all possible weight vectors with a uniform prior and employing a sign transfer function, the authors analytically derive a closed-form covariance function. This results in a Gaussian process with a specific kernel: $k(x, y) = 1 - \theta ||x - y||$. A unique property of this "sign kernel" is that the weight decay factor mathematically cancels out, rendering the kernel entirely hyperparameter-free.
Key Insights
- Faster Convergence: Affine Neural Networks are proven to converge to target functions at a rate of $O(h^{-2})$, compared to $O(h^{-1})$ for standard networks with the same number of parameters.
- Gradient Flow: The continuous affine parametrization prevents the "saturation" common in
tanhor sigmoid networks. By forcing the network to traverse non-saturated zones between weights, gradients flow more reliably during optimization. - Robustness: The derived sign kernel balances local discrimination with global influence, allowing it to perform well across different datasets without the extensive cross-validation required for Gaussian kernels.
Experimental Results
The non-parametric sign kernel was evaluated against Gaussian kernels on the USPS (handwritten digits) and LETTERS (character recognition) datasets.
On the USPS dataset, the sign kernel achieved a test error of 4.07% without any tuning. In contrast, while a perfectly tuned Gaussian kernel ($\sigma=6$) achieved 3.58%, its performance was extremely sensitive to the choice of $\sigma$, with errors jumping as high as 58.29% for poorly chosen parameters. Similar results were observed on the LETTERS dataset, where the sign kernel (5.5% error) was highly competitive with the best-tuned Gaussian kernel (5.3%).
Conclusion
Continuous neural networks offer a compelling framework for both improving the optimization of finite networks and deriving robust non-parametric methods. The Affine Neural Network provides better theoretical bounds and more stable gradients, while the hyperparameter-free sign kernel offers a "plug-and-play" solution for kernel-based learning. Future work includes parametrizing the weight function $V$ using secondary neural networks and further exploring the theoretical properties of the $1 - \theta||x-y||$ kernel.
Summary updated