TL;DR
This paper investigates why simply increasing the size of neural networks doesn't always lead to better performance. The authors demonstrate that for large networks, standard gradient descent optimization struggles to effectively utilize the added capacity, leading to underfitting. This suggests that improving optimization methods is crucial for unlocking the full potential of deep learning on massive datasets.
What Problem Does This Paper Address?
Deep learning has achieved remarkable success across various domains like vision, language, and audio. A common intuition is that more data and larger models should lead to better performance. However, prior research has hinted at diminishing returns when scaling up neural network capacity, meaning that simply adding more parameters doesn't always translate to a proportional improvement in accuracy. This paper directly tackles this phenomenon, asking: is there an inherent optimization issue that prevents us from efficiently training larger networks, even when ample data is available? The authors hypothesize that this "waste of capacity" might be due to underfitting, where the optimization algorithm fails to fully exploit the network's potential.
Methods
The core of the paper's experimental setup involves training Multi-Layer Perceptrons (MLPs) with varying numbers of hidden units on the ImageNet LSVRC-2010 dataset. To isolate the effect of capacity, they keep other factors constant. The network architecture consists of a single hidden layer with a logistic sigmoid activation function, followed by a softmax output layer for classification. Weights are initialized using the Glorot and Bengio (2010) method, and classification layer parameters are initialized to zero.
Crucially, the authors focus on training error as a measure of how well the network is learning. They perform a grid search for hyper-parameters like learning rate and the number of hidden units, prioritizing those that minimize training error. This is a deliberate choice to investigate underfitting, as regularization techniques, which are common for preventing overfitting, would likely hinder the reduction of training error. They use a learning rate decay strategy: the learning rate is reduced by 5% if the training error increases after an epoch. Experiments are conducted on a cluster of GPUs using the Theano library.
Datasets
The primary dataset used in this study is the ImageNet LSVRC-2010 dataset. This dataset is described as containing 1000 object categories and 1.2 million training images. The authors utilize pre-computed features for this dataset, which are derived from convolving images with SIFT features and then using K-Means to create a visual vocabulary of 1000 visual words. They report the Top-5 error rate, a standard metric for this challenge.
Key Insights and Ideas
- Diminishing Returns on Capacity are Real (and Problematic): The experiments clearly show that as the number of hidden units increases, the reduction in training error slows down significantly. This suggests that the network isn't effectively using the extra capacity to learn the training data better.
- Underfitting, Not Overfitting, is the Culprit: While practitioners often worry about overfitting with large models, this paper points to underfitting as the primary issue. The network's inability to reduce training error with increased capacity indicates it's not learning the training data sufficiently.
- First-Order Gradient Descent Struggles: The authors hypothesize that the failure to leverage added capacity is a consequence of first-order gradient descent optimization methods. They suggest that as networks grow, interactions between units can lead to an ill-conditioned Hessian matrix, making it difficult for simple gradient descent to find optimal solutions.
- A Baseline for Comparison: The introduction of a simple "template matcher" baseline (where an additional unit tries to match one of the training errors) provides a crucial reference point. The fact that the MLP fails to outperform this simple baseline after a certain capacity suggests a fundamental limitation in its learning process.
Key Results
- Training Error Plateaus: Figure 1 shows that while training error decreases with increasing hidden units, the rate of decrease slows dramatically. For instance, increasing capacity from 1000 to 15000 units results in a relatively small further reduction in training error.
- Return on Investment Plummets: Figure 2, which plots the "return on investment" (errors removed per additional unit), reveals a stark decline. Increasing capacity from 1000 to 2000 units leads to an order of magnitude decrease in this metric.
- MLP Falls Behind a Simple Baseline: The MLP's performance (errors removed per unit) drops below a simple template matching baseline after approximately 5000 hidden units. This is a significant finding, indicating that the complex optimization process of the MLP is less effective than a basic strategy for utilizing added capacity beyond a certain point.
- Learning Curves Converge: Figure 3 shows that for capacities above 5000 units, the learning curves (training error over epochs) converge to a similar point, further supporting the idea that additional capacity isn't leading to better learning.
Limitations and Future Work
Limitations:
- Focus on 1-Layer Networks: The experiments are primarily conducted on a single-layer sigmoidal neural network. The authors acknowledge that the problem of optimization failure might worsen for deeper networks, and their experimental setup should be extended to measure the effect of depth.
- Specific Optimization Method: The paper focuses on first-order gradient descent. While this is a common optimization method, other approaches might behave differently.
Future Work:
The authors propose two main research directions to address the observed optimization failure:
- Methods to Break Interactions: Developing techniques that can break interactions between large numbers of units could lead to a better-conditioned Hessian, improving the effectiveness of first-order descent. Examples include sparsity and orthogonality penalties.
- Methods to Model Interactions: Exploring second-order methods or natural gradient methods that can better model the complex interactions within large networks. The challenge here is scaling these methods to large datasets, where stochastic gradient approaches are often preferred.
- Distinguishing Ill-Conditioning from Local Minima: Future work should investigate methods to differentiate between issues caused by ill-conditioning of the Hessian and problems arising from the presence of many local minima in the loss landscape of large networks.
Summary updated