Selective Deep Autoencoder for Unsupervised Feature Selection

Wael Hassanieh, Abdallah A. ChehadeAAAI 2024 · DOI

TL;DR

This paper introduces Selective Deep Autoencoder (SDAE), a novel unsupervised feature selection method that leverages deep autoencoders to learn a representative subset of features without requiring prior knowledge of the number of features to select. SDAE effectively disentangles complex, non-linear feature relationships, leading to improved reconstruction and downstream task performance compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

In the era of big data, high-dimensional datasets are ubiquitous. While these datasets can boost the performance of data-driven models, they also introduce challenges such as high computational costs, increased risk of overfitting, and reduced model interpretability, often due to the presence of redundant features. Feature selection aims to address these issues by identifying a smaller, informative subset of features. However, unsupervised feature selection, which doesn't rely on labels, faces the challenge of identifying important feature subsets without explicit guidance and effectively handling inter-feature correlations. Existing methods often struggle with complex, non-linear relationships and require pre-specifying the number of features to select, limiting their adaptability.

Methods

The proposed Selective Deep Autoencoder (SDAE) framework tackles unsupervised feature selection in two main stages:

  1. Pre-training (Learning Latent Representations): A standard deep autoencoder is trained to minimize reconstruction error. This stage focuses on learning a high-quality, lower-dimensional latent representation of the input data. The encoder maps the input features to this latent space, and the decoder reconstructs the original features from these encodings. This process ensures that the latent representations capture essential information from the original feature space.
  2. Feature Selection (Learning Feature Subset): In this stage, the pre-trained encoder's architecture and weights are transferred. A novel "Selective Layer" is introduced, which consists of weights associated with each input feature. These weights are constrained to be non-negative and are regularized using an L1 norm to encourage sparsity (driving weights of irrelevant features to zero). The network is trained to reconstruct the latent representations (obtained from the pre-trained encoder) using only the selected features. This means the objective is to learn a subset of features that can accurately represent the learned latent space. The L1 regularization on both the encoder's layers and the Selective Layer's weights helps to prevent exploding weights and ensures that features are truly selected based on their contribution to reconstructing the latent space.

Crucially, SDAE intrinsically learns the optimal number of features to select, rather than requiring it as a hyperparameter.

Datasets

The paper evaluates SDAE on:

  • Simulated Dataset: A custom dataset designed to test SDAE's ability to handle linearly and non-linearly correlated features. It includes features derived from transformations like rotations and translations, as well as non-linear mappings.
  • Real Datasets:
    • Mice Protein dataset
    • ISOLET dataset
    • MNIST dataset

Key Insights and Ideas

  • Deep Architecture Advantage: SDAE's use of a deep autoencoder architecture, as opposed to the shallow architectures common in many existing AE-based feature selection methods, is crucial for disentangling complex, non-linear feature relationships. This is particularly beneficial for high-dimensional and intricate datasets.
  • Learning Latent Representations First: Separating the learning of latent representations from feature selection allows each stage to focus on its core task, leading to a more robust and effective overall approach.
  • Intrinsic Feature Count Selection: The ability of SDAE to automatically determine the optimal number of features to select is a significant advantage, removing a common limitation of other methods that require this parameter to be pre-specified.
  • Selective Layer with L1 Regularization: The novel Selective Layer, combined with L1 regularization, effectively drives the weights of irrelevant features to zero, thereby identifying a parsimonious and informative feature subset.

Key Results

  • Simulated Data: SDAE outperformed the Fractal Autoencoder (FAE) on a simulated dataset with non-linear features. For linear features, SDAE automatically selected one feature (k=1), matching the optimal for FAE, and offered comparable results. When all features were included, SDAE selected 4 features and achieved a significantly lower reconstruction error (2.32e-3) compared to FAE (0.9951 for k=3, and even worse for k=5).
  • Real Data: SDAE demonstrated strong performance across three real-world datasets (Mice Protein, ISOLET, MNIST).
  • Reconstruction: SDAE achieved one of the top 3 smallest reconstruction errors among baseline methods on all three datasets.
  • Classification: SDAE showed comparable downstream classification accuracy to other methods. Notably, on the Mice Protein dataset, where classification was challenging, SDAE achieved a superior accuracy of 94.4%, outperforming baselines like CAE (66.9%) and FAE (87.8%).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the optimization problem in feature selection can be NP-hard, and SDAE provides a sub-optimal solution.
  • While SDAE outperforms FAE on non-linear simulated data, the paper doesn't extensively explore the theoretical limits of its performance on extremely complex non-linear relationships.
  • Future Work:
    • The paper suggests that SDAE's deep structure allows for compression of quality information in the encodings, which could be further explored for other applications.
  • The authors' focus on unsupervised feature selection could be extended to semi-supervised or supervised settings.
  • Further investigation into the interpretability of the learned latent space and the selected features could be beneficial.

Summary updated

BibTeX
@inproceedings{hassanieh2024selective,
  author = {Wael Hassanieh and Abdallah A. Chehade},
  title = {{Selective Deep Autoencoder for Unsupervised Feature Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29123},
}
Figure