Recovery of linear components: Reduced complexity autoencoder designs

Federico Zocco, Seán F. McLooneEngineering Applications of AI 2020 · DOI · OpenReview

TL;DR

This paper introduces Recovery of Linear Components (RLC), a novel method that bridges the gap between linear and non-linear dimensionality reduction techniques. RLC significantly speeds up autoencoder training while maintaining or improving accuracy, making it practical for time-sensitive applications like semiconductor manufacturing optimization.

What Problem Does This Paper Address?

In data analysis, reducing the number of variables (dimensionality reduction and variable selection) is crucial to combat the "curse of dimensionality," improve model performance, and reduce computational costs. While linear methods like PCA are fast but struggle with non-linear data, traditional autoencoders excel at capturing non-linear relationships but are computationally expensive and time-consuming to train. This creates a need for methods that offer the best of both worlds: the ability to handle non-linearities without the prohibitive training burden.

Methods

The proposed Recovery of Linear Components (RLC) method offers a middle ground. It involves a two-stage process:

  1. Linear Encoding: First, a linear technique (like PCA for dimensionality reduction or FSCA for variable selection) is used to extract a set of klin linearly transformed components.
  2. Non-linear Decoding: Then, a neural network is trained to reconstruct the original data from a subset of these k (where k < klin) linear components. This means the neural network only needs to learn the non-linear relationships between the retained linear components and the discarded ones, rather than the entire complex mapping from raw input to reconstruction.

This approach leads to significantly less complex neural networks compared to full autoencoders, thus accelerating training. The paper also introduces two benchmark algorithms for comparison: PCA-SAE (PCA followed by a Stacked Autoencoder) and FSCA-SDE (FSCA followed by a Stacked Decoder), which represent more complex non-linear approaches.

Datasets

The authors evaluated their RLC method on:

  • Synthetic Dataset (Xsynthetic): A custom-generated dataset designed to test non-linear relationships.
  • Real-world Datasets:
    • Xwafer (semiconductor manufacturing optimization)
    • Xchemistry (chemical sensor array data)
    • Xbusiness (weekly product purchases)
    • YaleB (extended Yale face database B)
    • USPS (handwritten digit dataset)
    • Xsemicon (semiconductor manufacturing)

Key Insights and Ideas

  • Bridging the Gap: RLC successfully bridges the gap between fast but limited linear methods and powerful but slow non-linear autoencoders.
  • Computational Efficiency: By using a linear encoding step and a reduced non-linear decoding step, RLC drastically reduces the complexity of the neural network, leading to much faster training times.
  • Importance of Pre-training: For complex Stacked Autoencoders (SAEs), greedy layer-wise pre-training is crucial for achieving good performance and robustness. Without it, SAEs can be outperformed by simpler methods.
  • RLC's Robustness: RLC demonstrates robust performance across various datasets, achieving high variance explained with fewer components than linear methods, and doing so without the need for complex pre-training.
  • Practical Application: The method shows particular promise in time-sensitive applications like semiconductor manufacturing, where computational efficiency is paramount.

Key Results

  • Accuracy: RLC achieves higher accuracy than purely linear techniques and comparable or better accuracy than autoencoders of similar complexity.
  • Speed: RLC is significantly faster to train than conventional Stacked Autoencoders (SAEs). For example, PCA-SAE can take 30 to 400 times longer to train than PCA-RLC.
  • State-of-the-Art Performance: For the Xwafer dataset (semiconductor manufacturing optimization), FSCA-RLC outperforms the current state-of-the-art methods (SPBR and MPBR) in terms of achieving target reconstruction accuracy with fewer measurement sites, and is significantly faster to train than FSCA-SDE.
  • Robustness: RLC shows similar robustness to overfitting as more complex autoencoder methods.

Limitations and Future Work

  • Limitations:
    • The paper notes that for very high compression ratios (small k), PCA-SAE can outperform PCA-RLC due to its deeper non-linear architecture.
  • While RLC is faster than SAEs, it is still more computationally intensive than basic linear methods like FSCA.
  • Future Work: The paper implicitly suggests that RLC's architecture and training efficiency make it a strong candidate for real-time or near-real-time applications where traditional deep learning methods are too slow. Further exploration of RLC's application in other domains requiring efficient dimensionality reduction and variable selection would be a natural next step.

Summary updated

Abstract

Reducing dimensionality is a key preprocessing step in many data analysis applications to address the negative effects of the curse of dimensionality and collinearity on model performance and computational complexity, to denoise the data or to reduce storage requirements. Moreover, in many applications it is desirable to reduce the input dimensions by choosing a subset of variables that best represents the entire set without any a priori information available. Unsupervised variable selection techniques provide a solution to this second problem. An autoencoder, if properly regularized, can solve both unsupervised dimensionality reduction and variable selection, but the training of large neural networks can be prohibitive in time sensitive applications. We present an approach called Recovery of Linear Components (RLC), which serves as a middle ground between linear and non-linear dimensionality reduction techniques, reducing autoencoder training times while enhancing performance over purely linear techniques. With the aid of synthetic and real world case studies, we show that the RLC, when compared with an autoencoder of similar complexity, shows higher accuracy, similar robustness to overfitting, and faster training times. Additionally, at the cost of a relatively small increase in computational complexity, RLC is shown to outperform the current state-of-the-art for a semiconductor manufacturing wafer measurement site optimization application.
BibTeX
@article{zocco2020recovery,
  author = {Federico Zocco and Seán F. McLoone},
  title = {{Recovery of linear components: Reduced complexity autoencoder designs}},
  journal = {Engineering Applications of AI},
  year = {2020},
  doi = {10.1016/j.engappai.2022.104663},
}
Figure