Abstract
We investigate unsupervised pre-training of deep architectures as feature generators for "shallow" classifiers. Stacked Denoising Autoencoders (SdA), when used as feature pre-processing tools for SVM classification, can lead to significant improvements in accuracy—however, at the price of a substantial increase in computational cost. In this paper, we create a simple algorithm which mimics the layer-by-layer training of SdAs. However, in contrast to SdAs, our algorithm requires no training through gradient descent as the parameters can be computed in closed-form. It can be implemented in less than 20 lines of MATLAB and reduces the computation time from several hours to mere seconds. We show that our feature transformation reliably improves the results of SVM classification significantly on all our datasets—often outperforming SdAs and even deep neural networks in three out of four deep learning benchmarks.
Introduction
Deep learning architectures typically rely on a two-stage process: unsupervised pre-training to initialize weights followed by supervised fine-tuning via back-propagation. While effective, this approach suffers from high computational costs—pre-training often accounts for 50-90% of total training time—and extreme hyperparameter sensitivity. Traditional autoencoders require careful tuning of learning rates, momentum, and network architecture, often necessitating exhaustive cross-validation.
This paper introduces SLIDE (Stacked Linear Denoiser), a method that replaces computationally expensive unsupervised pre-training with a simple, closed-form linear algorithm. We explore whether the feature-engineering benefits of "deep" architectures can be achieved without the optimization complexities of non-linear gradient descent.
Methods: Stacked Linear Denoisers
The SLIDE algorithm simplifies feature learning through two primary components:
1. Linear Denoiser (LIDE)
Instead of a non-linear autoencoder, we utilize a linear regression model to reconstruct data from corrupted versions where features are randomly set to zero. Crucially, we derive a closed-form solution for the weights ($W$) in the limit of "infinite" noise realizations. This "Virtual Denoising" allows us to compute optimal transformations using simple matrix inversion rather than iterative training.
2. Stacking and Non-linearity
To capture complex, high-level features, these linear layers are stacked. Between each layer, a simple thresholding function ($T(a) = 1$ if $a > t$, else $0$) is applied. This non-linearity prevents the stack from collapsing into a single linear operation, allowing for the emergence of deep feature hierarchies.
The final features generated by these layers are concatenated and fed into a Support Vector Machine (SVM) with an RBF kernel. To handle varying feature scales across layers, we learn specific kernel widths for each layer's features.
Experimental Evaluation
We evaluated SLIDE on four prominent deep learning benchmarks: MNIST (handwritten digits), Rectangles (shape orientation), Rectangles-Images (noisy background rectangles), and Convex (convex region detection).
Performance and Speed
The most striking result is the massive reduction in training time. SLIDE reduces feature generation from hours to seconds. On the MNIST dataset, SLIDE achieved a speedup of approximately 1,377x compared to a GPU-optimized SdA implementation.
Despite its simplicity, SLIDE-SVM delivered competitive accuracy, outperforming standard SdA and even deep neural networks with back-propagation on three out of the four benchmarks (Rectangles, Rect-imgs, and Convex).
Discussion and Conclusion
SLIDE demonstrates that the "magic" of denoising autoencoders does not strictly require complex non-linear hidden units during feature discovery; linear mappings effectively capture essential feature correlations. By leveraging the Law of Large Numbers for "Virtual Denoising," we bypass the need to generate noisy samples, leading to a mathematically exact and highly efficient solution.
While SLIDE removes parameters like learning rates and epochs, it still requires cross-validation for noise levels and layer counts. Future work may explore alternative squashing functions or the application of SLIDE to non-spatial data types. Nevertheless, the ability to implement a high-performing deep feature learner in fewer than 20 lines of code represents a significant step toward more accessible and efficient deep learning.
Summary updated