Online Incremental Feature Learning with Denoising Autoencoders

Guanyu Zhou, Kihyuk Sohn, Honglak LeeAISTATS 2012 · OpenReview

Abstract

While determining model complexity is an important problem in machine learning, many feature learning algorithms rely on cross-validation to choose an optimal number of features, which is usually computationally expensive. This paper introduces an incremental learning algorithm for Denoising Autoencoders (DAEs) that automatically determines the optimal number of features during online training. By dynamically adding features to capture unexplained data patterns and merging redundant ones to prevent overfitting, the model adapts to massive, non-stationary data streams without the need for expensive cross-validation.

Introduction

In traditional machine learning, "model complexity"—specifically the number of features or hidden units—is a hyperparameter typically tuned using cross-validation. However, this approach fails in two major scenarios:

  1. Massive Online Data: When data arrives in a continuous, high-volume stream, performing multiple passes for cross-validation is computationally prohibitive.
  2. Non-Stationary Distributions: In real-world environments, the "rules" of the data often change over time (e.g., new classes appear or the ratio of labels shifts). A fixed-size model that was optimal at the start of training may become under-powered or unnecessarily bloated later on.

The authors seek a way to let the model "grow" or "shrink" its feature set on the fly, responding directly to the difficulty of the incoming data.

Methodology

The authors use the Denoising Autoencoder (DAE) as their building block. Their incremental algorithm revolves around two core processes:

Overview of the incremental learning process, illustrating the addition of new features to handle residuals and the merging of similar features to maintain compactness.
  • Adding Features: The model monitors its performance. "Hard" examples (those with high reconstruction or classification error) are collected into a buffer. Once the buffer reaches a threshold, the model greedily initializes new features specifically designed to minimize the "residual" error that the existing features couldn't handle.
  • Merging Features: To prevent the model from growing indefinitely and overfitting, the algorithm identifies pairs of features that are highly similar (using cosine distance). These are merged into a single feature, keeping the representation compact.

The training can be guided by three types of objective functions: Generative (minimizing reconstruction error), Discriminative (minimizing classification error), or a Hybrid of both. After adding or merging, the entire network is fine-tuned using standard gradient descent.

Experimental Setup

The researchers evaluated their approach on large-scale versions of standard benchmarks, scaled up to 1 million samples each to simulate "big data" conditions:

  • MNIST Variations: Including rot-1M (rotated digits), bg-img-1M (digits over complex image backgrounds), and bg-rand-1M (random noise backgrounds).
  • Rect-images: A dataset of rectangles with varying aspect ratios against different backgrounds.
  • Non-Stationary Stream: A custom version of the bg-img-1M dataset where the class distribution fluctuates over time according to a Gaussian Process.
Samples from the bg-img-1M dataset, featuring MNIST digits superimposed on complex natural image backgrounds.

Discussion

  • Dynamic Decoding Bias: When adding new features, the authors treat the output of the existing features as a "dynamic bias." This allows the new features to focus exclusively on the parts of the signal the model hasn't learned yet, making the incremental training highly efficient.
  • Merging as Regularization: Merging isn't just a compression trick; it acts as a form of regularization. By forcing redundant features together, the model maintains a more generalized representation of the data.
  • Self-Correction: One of the most interesting findings is that the model converges to a similar "optimal" number of features regardless of whether it starts with 400 or 800 units. It effectively "finds" the complexity inherent in the dataset.
  • Handling "Concept Drift": Because the model can add features at any time, it is far more robust to non-stationary data. When the data distribution shifts, the error spikes, triggering the model to add new features to accommodate the new patterns.

Results

  • Superior Accuracy: On the bg-img-1M task, the 1-layer incremental model (IncMDAE) achieved a classification error of 6.37%, significantly outperforming the standard DAE (7.65%) and linear SVM (29.24%).
  • Deep Learning Performance: When stacked into deep architectures, the incremental approach (IncMDAE$^{l1-3}$) outperformed standard Deep Belief Networks (DBNs) and Stacked Denoising Autoencoders (SDAEs) across almost all benchmarks.
  • Robustness to Change: In tests with fluctuating label distributions, the incremental model maintained a much lower and more stable error rate compared to static models, which struggled to adapt when the "mix" of digits changed.

Conclusion

While the algorithm demonstrates strong performance, it relies on specific heuristic thresholds (like the size of the "hard example" buffer and the error threshold $\mu$) which may require adjustment for different domains. Additionally, the logic for monitoring error and calculating feature similarity adds computational overhead compared to standard SGD. Future work could explore applying similar incremental logic to other architectures like Convolutional Neural Networks (CNNs) or Transformers.

Summary updated

Abstract

While determining model complexity is an important problem in machine learning, many feature learning algorithms rely on cross-validation to choose an optimal number of features, which is usually i...
BibTeX
@inproceedings{zhou2012online,
  author = {Guanyu Zhou and Kihyuk Sohn and Honglak Lee},
  title = {{Online Incremental Feature Learning with Denoising Autoencoders}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=34aolDTGZ1},
}
Figure