EigenNet: Towards Fast and Structural Learning of Deep Neural Networks

Ping LuoIJCAI 2017 · DOI

TL;DR

This paper introduces EigenNet, a novel deep neural network architecture that accelerates training and improves generalization by explicitly whitening both input features and gradients. By adapting techniques from natural gradient descent and incorporating a pruning mechanism, EigenNet effectively conditions the Fisher Information Matrix, leading to faster convergence and reduced overfitting.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are powerful but notoriously difficult to train and prone to overfitting. The difficulty in training stems from the "inhomogeneous curvature" of the loss function, which makes standard optimization methods like Stochastic Gradient Descent (SGD) inefficient, requiring careful tuning of learning rates for each layer. Overfitting, on the other hand, occurs when the network learns noise in the training data, leading to poor performance on unseen data. While methods like dropout help with overfitting and techniques like adaptive learning rates aim to speed up training, there's a need for a unified approach that addresses both issues effectively.

Methods

EigenNet introduces a new architecture that wraps existing DNN layers with two additional intermediate layers. These layers perform an "eigen-construction" process that involves:

  • Whitening Input Features and Gradients: Unlike previous methods that only whitened input features, EigenNet explicitly whitens both the forward pass (input features) and the backward pass (gradients). This is achieved by projecting the data and errors into an eigenspace derived from their covariance matrices.
  • Pruning Neurons: EigenNet incorporates a mechanism to prune "noisy" eigenvectors that have small eigenvalues. This effectively reduces the number of hidden neurons and helps to prevent overfitting by focusing on the most significant features.
  • Structural Learning: The eigen-construction process is performed periodically during training, allowing the network to adapt its structure and conditioning based on the evolving data distribution.
  • Gradient Shrinking: To stabilize training and prevent exploding gradients, a gradient shrinking mechanism is applied, especially when the initial learning rate is high.

Datasets

The paper evaluates EigenNet on the following datasets:

  • MNIST: Used for an ablation study and initial comparisons.
  • CIFAR-10: Used to evaluate compatibility with existing architectures.
  • CIFAR-100: Used to evaluate compatibility with existing architectures.
  • Image Reconstruction (Auto-encoder): A custom setup for evaluating generative unsupervised learning.

Key Insights and Ideas

  • Unified Approach to Training and Generalization: EigenNet elegantly tackles both training speed and overfitting by leveraging the principles of natural gradient descent and incorporating a structural adaptation mechanism.
  • Importance of Whitening Both Forward and Backward Passes: The paper demonstrates that whitening both input features and gradients is crucial for achieving significant improvements in both supervised and unsupervised learning tasks.
  • Pruning as a Structural Regularizer: The ability to prune noisy eigenvectors provides an effective way to reduce overfitting and automatically adapt the network's capacity without explicit hyperparameter tuning for network size.
  • Connection to Natural Gradient Descent: EigenNet can be seen as a practical implementation that approximates the benefits of natural gradient descent, which is known for its fast convergence but is computationally expensive.

Key Results

  • Faster Training: EigenNet significantly reduces training wall-clock time and the number of updates required to reach a certain accuracy compared to SGD, RMSprop, and even Batch Normalization. For example, on MNIST, EigenNet-FP reduced runtime by more than 5 times compared to PRONG+BN and 11 times compared to BN.
  • Improved Generalization: EigenNet variants consistently outperform baseline methods in terms of classification error on test sets, indicating reduced overfitting.
  • Effective for Unsupervised Learning: In generative tasks (auto-encoders), EigenNet-FP+BP achieved the best performance, demonstrating its potential beyond supervised learning.
  • Structural Adaptation: The pruning mechanism effectively reduced the number of neurons in layers, adapting the network structure to better capture discriminative patterns.

Limitations and Future Work

  • Computational Overhead: While EigenNet speeds up overall training, the eigen-construction step itself introduces some computational overhead per iteration. The paper notes that the runtime of each feed-forward iteration for EigenNet-FP is about 2x that of SGD, but the progress made per update compensates for this.
  • Hyperparameter Tuning for Gamma: The paper mentions that the whitening interval gamma can affect performance, and while it explores different values, further research into optimal gamma selection could be beneficial.
  • Future Work: The authors suggest applying EigenNet to other generative models like GANs and LSTMs, and exploring regularization of non-diagonal FIM blocks to further reduce co-adaptation between hidden layers.

Summary updated

BibTeX
@inproceedings{luo2017eigennet,
  author = {Ping Luo},
  title = {{EigenNet: Towards Fast and Structural Learning of Deep Neural Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/338},
}
Figure