Fractal Autoencoders for Feature Selection

Xinxing Wu, Qiang ChengAAAI 2021 · DOI

Introduction

In the era of high-throughput data collection, researchers are frequently confronted with datasets containing thousands of dimensions, from high-resolution imagery to complex genomic sequences. However, much of this data is often redundant or noisy, complicating downstream analysis and increasing computational costs. Unsupervised feature selection aims to identify a compact, informative subset of features that preserves the essential characteristics of the original data without requiring labeled training sets.

Existing unsupervised methods often struggle with two primary issues: the vast search space of potential feature subsets and the tendency to select redundant features that are individually important but collectively provide little new information. To address these challenges, researchers have introduced the Fractal Autoencoder (FAE), a novel deep learning framework designed to identify both informative and diverse feature subsets.

The Fractal Autoencoder Framework

The FAE architecture evolves from the standard autoencoder (AE) through a series of structural innovations designed specifically for feature selection. A standard AE compresses data into a bottleneck layer and then attempts to reconstruct the original input. While effective for dimensionality reduction, it does not explicitly select features from the original input space.

To enable feature selection, the authors first developed the Identification Autoencoder (IAE) by adding a "one-to-one" scoring layer at the input. In this layer, each input feature is assigned a weight; features with weights near zero are effectively ignored. However, the IAE still lacks a mechanism to penalize redundancy among high-scoring features.

The FAE architecture featuring a dual-path structure: a primary network using all weighted features and a sub-network (fractal path) restricted to the top-k features.

The "fractal" innovation in FAE introduces a dual-path training objective. As shown in the architecture diagram, the model consists of two parallel paths:

  1. The Primary Path: Uses all features (weighted by the one-to-one layer) to reconstruct the data.
  2. The Sub-network (Fractal Path): Is forced to reconstruct the entire dataset using only the top-k highest-scoring features identified by the scoring layer.

By training these paths jointly, the network is incentivized to ensure that the top-k features are not just individually significant, but collectively sufficient to represent the whole dataset. This self-similar structure—where a small part of the network performs the same task as the whole—is the inspiration for the "fractal" designation.

Experimental Evaluation and Key Insights

The FAE was validated across 14 diverse real-world datasets, including image recognition (MNIST, COIL-20), biological studies (Mice Protein, various Gene Expression datasets), and sensor data (Activity).

A central insight from this work is that diversity is as critical as importance. Traditional methods might select 50 highly correlated pixels from the same region of an image, wasting capacity. FAE’s sub-network explicitly penalizes such redundancy. Furthermore, FAE offers superior stability compared to probabilistic methods like Concrete Autoencoders (CAE), which can be difficult to train on extremely high-dimensional data.

Results and Impact

FAE consistently achieved state-of-the-art performance, yielding lower reconstruction errors and higher downstream classification accuracy than competitors like CAE and AgnoS.

Visual Evidence on MNIST

Visual analysis of the features selected from the MNIST dataset provides clear evidence of FAE's effectiveness. While simpler methods often select clusters of redundant pixels, FAE identifies pixels that outline the "skeleton" of the handwritten digits, capturing the most diverse and representative structural information.

Visual comparison of feature selection methods on MNIST. FAE (d, f, h) identifies a more diverse and representative 'skeleton' of the digits compared to the more redundant selections of IAE (c, e, g).

Efficiency in Genomics

In practical applications, FAE demonstrated significant economic value. When applied to a massive Gene Expression Omnibus (GEO) dataset, FAE identified a subset of 800 genes capable of reconstructing the data with the same accuracy as the industry-standard "L1000" landmark set (which requires 943 genes). This represents a 15% reduction in measurement costs for large-scale biological studies.

Conclusion and Future Work

The Fractal Autoencoder provides a robust, deterministic, and effective framework for unsupervised feature selection. While the dual-path training increases computational cost by approximately a factor of two compared to standard sparse autoencoders, the gains in feature diversity and reconstruction accuracy are substantial. Future research will explore extending this fractal framework to other machine learning tasks and investigating more complex nonlinear architectures for the initial feature selection layer.

Summary updated

BibTeX
@inproceedings{wu2021fractal,
  author = {Xinxing Wu and Qiang Cheng},
  title = {{Fractal Autoencoders for Feature Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i12.17242},
}
Figure