A new Stack Autoencoder: Neighbouring Sample Envelope Embedded Stack Autoencoder Ensemble Model

Chuanyan Zhou, Jie Ma, Fan Li, Yongming Li, Pin Wang, Xiaoheng ZhangarXiv 2022 · DOI · arXiv

Abstract

Standard Stacked Autoencoders (SAEs) often fail to capture the intrinsic relationships between data points, leading to suboptimal feature representation. This paper introduces the Neighbouring Sample Envelope Embedded Stack Autoencoder Ensemble (NE_ESAE), a novel framework that shifts the focus from isolated data samples to the structural relationships between neighboring points. By utilizing a Neighbouring Sample Envelope Learning Mechanism (NSELM) and an embedded ensemble architecture, the model achieves significant performance gains, including a 40% accuracy improvement on complex medical datasets compared to traditional SAEs.

Introduction: The "Tunnel Vision" of Standard SAEs

Stacked Autoencoders are widely used for unsupervised feature learning. However, they typically suffer from a "tunnel vision" problem: they process each data sample in isolation. By focusing exclusively on minimizing the reconstruction error of individual points, these models ignore the hierarchical structural information—the intrinsic similarities and relationships between different samples in a dataset.

Figure 3: Traditional autoencoders process samples independently, failing to account for the structural relationships between neighboring data points (i and j).

This lack of structural awareness leads to poor separability in the feature space, particularly when dealing with complex, non-linear data. To address this, the authors argue that a model must consider "neighborhoods" of samples to truly understand the underlying data distribution.

Methodology

The NE_ESAE framework consists of three integrated stages designed to capture and preserve structural information:

  1. NSELM (Neighbouring Sample Envelope Learning Mechanism): This preprocessing step uses Sample-Pair Connection (SPC) to find a sample's nearest neighbor and concatenate them into an "envelope." This effectively doubles the feature space and forces the network to learn relationships between points. Iterative Mean Clustering (ICMC) is then used to generate multiple hierarchical sample spaces.
  2. ESAE (Embedded Stack Autoencoder): Unlike standard architectures, the ESAE embeds original feature information into every hidden layer. This ensures that the relationship between raw input and deep features is maintained throughout the training process, preventing feature drift.
  3. MSEM (Multilayer Space Ensemble Mechanism): The model treats different hierarchical sample spaces as an ensemble, fusing classification results through Weighted Fusion or Majority Voting to produce a final, stable prediction.

Results and Discussion

The model was validated on 12 diverse public datasets, including large-scale satellite data and high-dimensional medical records. The impact of the sample transformation on data separability is significant.

Figure 4: Comparison of feature space (a) before and (b) after the proposed sample transformation, demonstrating significantly improved class separability.

Key Performance Highlights:

  • Alzheimer’s Disease (AD) Dataset: NE_ESAE achieved 90.67% accuracy, a staggering improvement over the 50.67% achieved by standard SAEs.
  • Robustness: The ensemble approach proved more stable than single-layer feature extraction, particularly in datasets with small sample sizes and high dimensionality, such as LSVT Voice Rehabilitation.
  • Distribution Consistency: By using Maximum Mean Difference (MMD), the researchers ensured that statistical distributions remained consistent across hierarchical layers, preventing model confusion during the ensemble phase.

Conclusion

NE_ESAE represents a significant advancement in autoencoder design by prioritizing the structural context of data samples. While the current model is optimized for tabular and vector data, future work will focus on adapting the "envelope" approach for raw image pixels and speech waveforms, as well as developing a strictly end-to-end training architecture to reduce computational overhead.

Summary updated

BibTeX
@misc{zhou2022new,
  author = {Chuanyan Zhou and Jie Ma and Fan Li and Yongming Li and Pin Wang and Xiaoheng Zhang},
  title = {{A new Stack Autoencoder: Neighbouring Sample Envelope Embedded Stack Autoencoder Ensemble Model}},
  year = {2022},
  eprint = {2210.14956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.14956},
}
Figure