Do Deep Nets Really Need to be Deep?

Jimmy Ba, Rich Caruana

Deep neural networks have dominated machine learning, achieving state-of-the-art performance in complex tasks like speech and object recognition. But is their depth strictly necessary for representing complex functions, or does it merely make these functions easier to learn? This paper demonstrates that shallow, single-layer feed-forward networks can achieve accuracies comparable to deep convolutional models. However, they cannot easily learn these functions directly from raw data; instead, they must be trained to "mimic" a pre-trained deep model.

Representational Power vs. Learnability

The Universal Approximation Theorem states that a single hidden layer can represent any continuous function, given enough capacity. Yet, empirical results consistently show deep architectures outperforming shallow ones. The authors, Lei Jimmy Ba and Rich Caruana, investigate whether this gap is due to representational power (deep nets can represent functions shallow nets cannot) or learnability (current optimization algorithms struggle with shallow architectures but succeed with deep ones).

Methodology: Model Compression

To test this, the researchers used a technique known as Model Compression (or distillation). The process involves two main steps:

  1. Train a Teacher: A state-of-the-art deep model (or an ensemble of them) is trained on a labeled dataset.
  2. Train a Student (Mimic): A shallow network with a single hidden layer is trained to mimic the teacher's outputs.

Crucially, the student is not trained on the original 0/1 "hard" labels. Instead, it performs regression on the logits—the raw values produced by the teacher model before the final softmax activation. Logits provide a much richer training signal by preserving the relative relationships between classes (e.g., an image of a "dog" might have a higher logit for "cat" than for "car").

To handle the massive number of parameters required for a high-capacity shallow net, the authors introduced a linear bottleneck layer. By factorizing the large weight matrix into two lower-rank matrices, they significantly sped up training without reducing the network's theoretical representational capacity.

Key Results

The authors evaluated their approach on two benchmark tasks: the TIMIT speech corpus for phoneme recognition and the CIFAR-10 image recognition dataset.

TIMIT Phoneme Recognition: A shallow mimic network with 400k hidden units achieved a 20.0% Phone Error Rate (PER). This outperformed a standard deep neural network trained directly on the data (21.9%) and nearly matched the performance of a complex deep convolutional model (19.5%).

TIMIT Phone Error Rates

As shown in the performance curves below, the shallow mimic network (ShallowMimicNet) scales much better with increased parameters compared to a standard shallow network trained directly on the data, eventually approaching the accuracy of the deep convolutional teacher.

Accuracy on TIMIT Dev Set vs Parameters

CIFAR-10 Image Recognition: Using an additional 1 million unlabeled images from the 80 Million Tiny Images dataset to provide more data for the mimicry process, the shallow mimic net reached an error rate of 14.2%, which was comparable to deep convolutional architectures of that era.

Insights and Implications

  • Depth as a Training Aid: The results suggest that depth acts more as a "regularizer" or a structural hint that makes optimization easier, rather than a strict requirement for representing complex functions.
  • The Learnability Gap: Shallow nets can represent the complex functions learned by deep nets, but they are remarkably difficult to train directly on the original data. The deep model acts as a pedagogical bridge, smoothing the optimization landscape for the shallow student.
  • Parallelism vs. Sequentiality: Shallow models are inherently more parallelizable. While a deep net must process layers sequentially, a shallow net can finish its computation in fewer cycles, which is highly beneficial for low-latency applications.

Limitations and Future Work

The primary limitation of this approach is the "two-step" requirement. The authors were unable to train these high-performing shallow nets directly from the original labeled data; they still needed a deep "teacher" to find the function first.

Future work should focus on developing better optimization algorithms or regularization techniques that would allow us to train high-capacity shallow models directly. Additionally, the mimicry process works best when there is a large amount of unlabeled data available, which may not always be the case for every domain.

Summary updated

Abstract

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this paper we empirically demonstrate that shallow feed-forward nets can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow nets can learn these deep functions using the same number of parameters as the original deep models. On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional models.
BibTeX
@inproceedings{ba2014do,
  author = {Jimmy Ba and Rich Caruana},
  title = {{Do Deep Nets Really Need to be Deep?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=VPisCR90s8e},
}
Figure