Post Training in Deep Learning with Last Kernel

Thomas Moreau, Julien AudiffrenarXiv 2016 · OpenReview

TL;DR

This paper introduces "post-training," a simple yet effective method that further optimizes only the last layer of a deep neural network after its initial training. This technique consistently improves performance across various architectures and datasets by ensuring the learned data representations are utilized optimally for the specific task, offering a significant boost with minimal computational overhead.

What Problem Does This Paper Address?

Training deep neural networks is notoriously challenging due to complex, non-convex optimization landscapes, often leading to suboptimal solutions. While techniques like unsupervised pre-training can improve performance, there's a need to further refine how the learned representations are used for the final task. This paper addresses the gap of efficiently leveraging the rich feature representations learned by the initial layers of a deep network, aiming to maximize their utility for the specific downstream task without extensive retraining.

Methods

The core contribution is the "post-training" step. This method involves freezing all layers of a pre-trained neural network except for the last one, which is then further trained. This approach is motivated by the observation that early layers of a deep network learn general features, while later layers are task-specific. By only training the last layer, the method focuses on adapting the final decision-making component to the learned representation.

The authors also provide a theoretical grounding for this technique by connecting it to kernel theory. They show that the first $L-1$ layers can be viewed as a feature map, and the last layer's training becomes a kernel regression problem. This connection helps explain why post-training is effective and suggests that the learned features can act as a kernel. The post-training optimization problem is shown to be convex under common conditions (softmax activation and cross-entropy loss), ensuring efficient convergence.

Datasets

The paper evaluates the proposed post-training method on several standard datasets:

  • CIFAR10: A dataset of 60,000 images (32x32) for 10 object classes.
  • MNIST: A dataset of 65,000 images (27x27) for 10 classes.
  • FACES: A dataset of pre-processed images for 40 classes.
  • Penn Tree Bank (PTB): A dataset for language modeling, comprising 929k training words and 82k test words with a 10,000-word vocabulary.
  • Parkinson Telemonitoring data set: A real-world regression dataset with 5,875 instances and 17 features.
  • Simulated Data Set: A synthetic regression dataset with 10,000 samples.

Key Insights and Ideas

  • Separation of Concerns: Post-training elegantly separates the learning of general data representations (by early layers) from the task-specific adaptation (by the last layer). This is more efficient than end-to-end fine-tuning, which can sometimes lead to the general features being distorted.
  • Kernel Analogy: The connection to kernel methods provides a strong theoretical justification. It suggests that the early layers of a deep network can effectively act as a kernel function, transforming data into a feature space where the final task can be solved more easily.
  • Computational Efficiency: Post-training is significantly faster than traditional training or fine-tuning because it involves optimizing far fewer parameters and the optimization problem is often convex.
  • Robustness to Training Stage: The benefits of post-training are observed even when applied after the network has largely converged or even started to overfit, suggesting it's a valuable addition regardless of the initial training's completeness.

Key Results

  • Consistent Performance Boost: Across all tested architectures (convolutional, recurrent, and fully connected) and datasets, post-training consistently improved test performance (e.g., reduced error rates or perplexity).
  • Speedup: Post-training iterations are reported to be up to 4x faster than regular training iterations on the same GPU.
  • Minimal Data Requirement: The performance gains are achieved with as few as 100 post-training iterations, which is negligible compared to the overall training time.
  • No Overfitting: The method does not appear to cause overfitting; in fact, it often improves generalization, likely due to the reduced parameter space and added regularization. For example, on CIFAR10, post-training improved test error from 0.154 to 0.147 even after extensive regular training.

Limitations and Future Work

  • Focus on Last Layer: The current work focuses solely on training the last layer. The authors suggest that exploring post-training with more than just the last layer frozen could be an interesting direction for future research.
  • Kernel Solution Scalability: While the kernel perspective provides theoretical insights, the closed-form kernel solution is computationally infeasible for large datasets, highlighting the practical advantage of the proposed gradient-based post-training.
  • Regularization Parameter Tuning: The effectiveness of post-training is linked to the regularization parameter ($\lambda$). While the paper suggests a range ($10^{-5} < \lambda < 10^{-2}$), optimal tuning might still be required for specific tasks.

Summary updated

Abstract

One of the main challenges of deep learning methods is the choice of an appropriate training strategy. In particular, additional steps, such as unsupervised pre-training, have been shown to greatly improve the performances of deep structures. In this article, we propose an extra training step, called post-training, which only optimizes the last layer of the network. We show that this procedure can be analyzed in the context of kernel theory, with the first layers computing an embedding of the data and the last layer a statistical model to solve the task based on this embedding. This step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task. This idea is then tested on multiple architectures with various data sets, showing that it consistently provides a boost in performance.
BibTeX
@misc{moreau2016post,
  author = {Thomas Moreau and Julien Audiffren},
  title = {{Post Training in Deep Learning with Last Kernel}},
  year = {2016},
  eprint = {1611.04499},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.04499},
}
Figure