To go deep or wide in learning?

Gaurav Pandey, Ambedkar Dukkipati

Abstract

To achieve acceptable performance for AI tasks, one can either use sophisticated feature extraction methods as the first layer in a two-layered supervised learning model, or learn the features directly from data using deep architectures. This paper explores "wide learning," an approach that replaces deep, multi-layered architectures with a single hidden layer of infinite width. By using covariance arc-cosine kernels, the authors demonstrate that learning a distribution over features in one "infinitely wide" layer can outperform traditional deep belief networks and stacked autoencoders on several complex image recognition benchmarks.

Introduction

In the early 2010s, the dominant paradigm in machine learning suggested that "deep is better." Deep architectures, such as Deep Belief Networks (DBNs), were favored for their ability to extract hierarchical features. However, these models are often non-convex, prone to local optima, and require extensive hyperparameter tuning. This research investigates whether the perceived necessity of depth is actually a proxy for the need for greater "width"—specifically, a larger number of features. The authors propose that a shallow model, if made sufficiently wide, can capture complex invariances without the optimization challenges inherent in deep learning.

Methodology: Covariance Arc-Cosine Kernels

The core innovation is the Covariance Arc-Cosine Kernel, which provides a mathematical framework for "wide learning."

  1. Infinite Width: Arc-cosine kernels are mathematically equivalent to the computation of a neural network layer with an infinite number of hidden units. This allows the model to handle "width" analytically, bypassing the need to manually select the number of hidden units.
  2. Learning the Covariance ($\Sigma$): Unlike standard kernels with fixed parameters, the authors propose learning a specific covariance matrix for the weight distribution. This enables the model to identify the most important directions in feature space, effectively learning a distribution over an infinite set of features.
  3. Training Approaches: The authors explore both exact learning, using a stochastic gradient descent update rule for the covariance matrix, and inexact learning, which estimates the covariance matrix from the weights of a pre-trained finite-width Restricted Boltzmann Machine (RBM).

Efficiency and Convergence

A significant advantage of the infinite-width model is its speed of convergence. Because the "width" is handled analytically, the model reaches high accuracy in far fewer training iterations than finite-width counterparts.

Accuracy vs. Iterations: The wide model (red) achieves superior accuracy significantly faster than the finite-width model (blue).

As shown in the figure above, the wide model achieves approximately 98.9% accuracy on MNIST after only two iterations, whereas the finite-width model requires significantly more training to approach similar levels.

Empirical Evaluation

The authors evaluated the wide learning approach against several state-of-the-art deep models on challenging image recognition tasks, including rotated MNIST, MNIST with random backgrounds, and geometric classification tasks (Rectangles and Convex).

Comparative error rates across various image recognition benchmarks. The Covariance Arc-cosine kernel consistently outperforms deep architectures like DBNs and Stacked Autoencoders.

The results demonstrate that the single-layer wide model consistently outperforms deep architectures like DBNs and Stacked Autoencoders on nearly every benchmark. For instance, on the "MNIST rotated" dataset, the wide model achieved an 8.11% error rate compared to the DBN's 12.30%.

Discussion and Conclusion

The findings suggest that "wide learning" can make traditional backpropagation and fine-tuning stages redundant. By capturing a smooth and descriptive representation through an infinite number of features, the model provides a powerful alternative to hierarchical depth.

However, limitations remain. Kernel methods face $O(N^2)$ or $O(N^3)$ complexity relative to the number of training samples, posing scalability challenges for massive datasets. Furthermore, while the single-layer model is highly effective, the authors found that stacking multiple wide layers did not always yield improvements, suggesting that the optimal way to combine depth and infinite width is still an open question. Future work will focus on scaling these methods to more complex natural image datasets.

Summary updated

Abstract

To achieve acceptable performance for AI tasks, one can either use sophisticated feature extraction methods as the first layer in a two-layered supervised learning model, or learn the features dire...
BibTeX
@inproceedings{pandey2014go,
  author = {Gaurav Pandey and Ambedkar Dukkipati},
  title = {{To go deep or wide in learning?}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
  url = {https://openreview.net/forum?id=IDINoLPDkNX},
}
Figure