Abstract
To achieve acceptable performance for AI tasks, one can either use sophisticated feature extraction methods as the first layer in a two-layered supervised learning model, or learn the features directly using a deep (multi-layered) model. While the first approach is very problem-specific, the second approach has computational overheads in learning multiple layers and fine-tuning of the model. In this paper, we propose an approach called wide learning based on arc-cosine kernels, that learns a single layer of infinite width. We propose exact and inexact learning strategies for wide learning and show that wide learning with single layer outperforms single layer as well as deep architectures of finite width for some benchmark datasets.
Introduction
In the early 2010s, the prevailing wisdom in machine learning was that "deeper is better." However, training multi-layer neural networks presents significant challenges: models are non-convex, prone to local optima, and require extensive hyperparameter tuning. This paper challenges that paradigm by introducing Wide Learning. Instead of increasing depth, the authors explore whether a single, infinitely wide layer can match or exceed the representational power of deep networks while avoiding their optimization difficulties.
Methodology: The Covariance Arc-Cosine Kernel
The core of this approach is the Covariance Arc-Cosine Kernel, which builds upon the observation that as the number of hidden units in a random neural network approaches infinity, the inner product of hidden representations converges to a deterministic kernel function.
The authors move from learning specific weights for finite neurons to learning the covariance matrix ($\Sigma$) of a distribution from which infinite weights could be sampled. They derive a novel update rule by "kernelizing" the Contrastive Divergence algorithm used in Restricted Boltzmann Machines (RBMs), allowing for direct updates to $\Sigma$. Two strategies are proposed:
- Exact Method: Updating $\Sigma$ via gradient descent.
- Inexact Method: Training a standard RBM first and using its weights to estimate the covariance, providing a faster alternative.
This method essentially uses the kernel trick to perform unsupervised feature learning in an infinite-dimensional space, creating a "soft binning" effect that results in smoother and more robust data representations.
Experimental Results
The model was evaluated against several benchmarks, including MNIST (original, rotated, and with random backgrounds), as well as "Rectangles" and "Convex" shape discrimination tasks.
As shown in Figure 1, the infinite-width model demonstrates superior convergence speed and accuracy compared to finite-width models. While finite models struggle to reach peak performance even after many iterations, the Covariance Arc-cosine kernel achieves high accuracy almost immediately.
The comparative results in Figure 2 highlight the effectiveness of wide learning. On the original MNIST dataset, the model achieved a 0.95% error rate, surpassing standard Deep Belief Networks (DBNs). It also set new benchmarks on the "Rectangles" and "Convex" datasets, outperforming both DBNs and Stacked Autoencoders.
Discussion and Conclusion
The findings suggest that infinite width can serve as a powerful proxy for depth, capturing many of the invariances that typically require multiple layers. Notably, the authors found that with a sufficiently wide layer, the traditional "fine-tuning" stage of deep learning becomes largely redundant; a simple linear classifier on top of the unsupervised representation is often sufficient.
However, limitations remain. Like other kernel methods, wide learning faces a "scaling wall" due to $O(N^2)$ or $O(N^3)$ computational complexity relative to the number of training instances. Furthermore, stacking multiple infinite-width layers ("Deep-Wide" learning) surprisingly decreased performance in these tests, suggesting that for certain tasks, a single wide layer is optimal. Future work will focus on scaling these methods to larger datasets and exploring their applicability to non-image tasks like natural language processing.
Summary updated