Do Deep Nets Really Need to be Deep?

Lei Jimmy Ba, Rich Caruana

TL;DR

This paper challenges the prevailing notion that deep neural networks are inherently superior due to their depth. The authors demonstrate that shallow, fully-connected networks can achieve comparable performance to complex deep architectures on speech and image recognition tasks by mimicking the outputs of these deep models. This suggests that the "depth" advantage might be more about effective learning algorithms and regularization than an intrinsic need for hierarchical feature extraction.

What Problem Does This Paper Address?

The field of deep learning has seen remarkable success, with deep neural networks (DNNs) becoming the state-of-the-art in areas like speech recognition and computer vision. However, the question of why these deep architectures perform so well remains a subject of debate. Is it the depth itself, enabling hierarchical feature learning? Is it the specific architectural choices like convolutions? Or is it simply that current training algorithms and regularization techniques are better suited for deep networks? This paper specifically investigates whether the complexity and performance of deep models are truly tied to their depth, or if similar functions can be learned by shallower, simpler architectures.

Methods

The core of the paper's methodology is model compression, specifically training a "mimic" shallow neural network (SNN) to replicate the function learned by a pre-trained, more complex deep neural network (DNN). Instead of training the SNN on the original hard labels (0/1), they train it to predict the logits (pre-softmax outputs) of the teacher DNN. This approach, termed "mimic learning via regressing logit with L2 loss," is argued to provide richer information to the student model, capturing subtle relationships learned by the teacher.

To address the computational challenges of training very wide shallow networks, they introduce a linear bottleneck layer that factorizes the large input-to-hidden weight matrix. This speeds up training and reduces memory usage without sacrificing representational power, as the linear layer can be absorbed into the main weight matrix.

Datasets

  • TIMIT: A speech corpus used for phoneme recognition. The paper uses the standard training set (462 speakers), development set (50 speakers), and test set (24 speakers). Input features are derived from Mel-scale filter-banks.
  • CIFAR-10: An image recognition dataset consisting of 32x32 pixel images from 10 object classes. The paper uses the standard 50,000 training and 10,000 test images. They also augment the training data with images from the larger "80 million tiny images" dataset to create a larger transfer set.

Key Insights and Ideas

  • Depth is not always essential: The most striking finding is that shallow, fully-connected networks can achieve performance comparable to deep, complex architectures (like CNNs) when trained to mimic the deep model's outputs. This suggests that the "deep" advantage might be an artifact of current training methods rather than an inherent requirement for learning complex functions.
  • Mimicking logits is powerful: Training a shallow network to predict the logits of a deep network is more effective than training on original hard labels. This "soft" target provides more nuanced information, allowing the shallow network to learn finer details and relationships captured by the teacher.
  • Model compression as regularization: The mimic learning process itself acts as a form of regularization. By learning to match a teacher's output, the shallow network is guided and potentially prevented from overfitting as severely as it might when trained directly on the original data.
  • Training algorithms are key: The paper implies that the success of deep learning might be heavily influenced by the development of training algorithms and regularization techniques that work well with deep architectures. If similar algorithms were available for shallow networks, the performance gap might disappear.

Key Results

  • TIMIT Phoneme Recognition:
    • Shallow mimic nets (SNN-MIMIC-8k and SNN-MIMIC-400k) achieved phone error rates (PER) of 21.6% and 20.0% respectively.
  • These results are comparable to or better than deep models like DNN (21.9% PER) and CNN (19.5% PER), and even approach the performance of an ensemble of CNNs (ECNN, 18.5% PER).
  • Crucially, the SNN-MIMIC-400k, with no convolutional or pooling layers, achieved a PER of 20.0%, outperforming the DNN and nearly matching the CNN.
  • CIFAR-10 Object Recognition:
    • Shallow mimic nets (SNN-CNN-MIMIC-30k and SNN-ECNN-MIMIC-30k) achieved classification error rates of 15.4% and 14.2% respectively.
  • These results are comparable to deeper convolutional networks, with the SNN-ECNN-MIMIC-30k achieving an error rate close to the ensemble of deep CNNs.
  • Training Speed: Shallow mimic models trained significantly faster (1-2 hours) than deep models (8-12 hours) on comparable hardware, despite sometimes having more parameters.

Limitations and Future Work

  • Awkwardness of the Mimic Approach: The authors acknowledge that their mimic learning approach is "awkward." It relies on having a highly accurate teacher model or a large amount of unlabeled data to bridge the gap between the teacher and mimic model. This dependency makes it less straightforward than direct training.
  • Need for Better Training Algorithms: The paper strongly suggests that the primary limitation for shallow networks is not their capacity but the lack of effective learning and regularization algorithms. Developing algorithms to train shallow models directly from original data with high accuracy would be a significant contribution.
  • Future Work: The authors plan to investigate labeling the 80 million tiny images dataset to train shallow models without convolutional layers to mimic deep convolutional models. They also aim to explore practical applications where large, high-accuracy deep models can be mimicked by smaller, faster student models.

Summary updated

Abstract

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.
BibTeX
@inproceedings{ba2014do,
  author = {Lei Jimmy Ba and Rich Caruana},
  title = {{Do Deep Nets Really Need to be Deep?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=VPisCR90s8e},
}
Figure