SOSAE: Self-Organizing Sparse AutoEncoder

Abstract

Traditional autoencoders require manual tuning of the bottleneck layer size, leading to significant computational overhead. We present SOSAE (Self-Organizing Sparse Autoencoder), which utilizes a novel "push loss" regularization to achieve structured sparsity. By forcing active neurons to one side of the feature vector, SOSAE enables automatic bottleneck optimization in a single training run, reducing hyperparameter tuning FLOPs by up to 130x while maintaining or improving accuracy across standard benchmarks.

Introduction

Unsupervised learning relies heavily on autoencoders for feature extraction and data compression. A persistent challenge is determining the optimal hidden layer size: undersized bottlenecks lose information, while oversized ones lead to computational waste and overfitting. Standard Sparse Autoencoders (SAEs) often produce unstructured sparsity, which offers limited practical benefits for hardware acceleration. SOSAE addresses these issues by introducing structured sparsity through a positional penalty, allowing the model to "prune" itself dynamically.

Methods

The core innovation of SOSAE is the Push Loss regularization. Unlike $L_1$ or $L_2$ penalties that encourage global sparsity, Push Loss applies a positional penalty that increases exponentially with the neuron's index in the feature vector. This "pushes" essential information into the initial dimensions, leaving the tail end of the vector inactive.

Expand

Source, p. 4

The model architecture consists of an encoder $g_\phi$ and a decoder $f_\theta$. The total loss function combines reconstruction error, the Push Loss, and a Contractive term based on the Frobenius norm of the Jacobian matrix to ensure feature robustness. This structured arrangement allows the model to simply clip inactive dimensions, effectively treating the bottleneck size as a trainable parameter rather than a fixed hyperparameter.

Results

SOSAE was evaluated on MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet. A key finding is the model's "information immunity"—the ability to maintain performance even after significant truncation of the feature vector.

Expand

Source, p. 6

On the MNIST dataset, SOSAE achieved a 76% reduction in memory usage (from 96 MB to 22.56 MB) while improving classification accuracy to 97.58%. Computationally, it reached optimal configurations with 130x fewer FLOPs than grid search. Furthermore, SOSAE demonstrated superior robustness in denoising tasks compared to traditional Denoising Autoencoders (DAE), maintaining higher accuracy across various Gaussian noise levels.

Conclusion

SOSAE provides a scalable solution to the bottleneck optimization problem in autoencoders. By organizing sparsity, it bridges the gap between theoretical regularization and practical computational savings. Future work will explore applying Push Loss to deep architectures and diverse data modalities like audio and video.

@misc{modi2025sosae,
  author = {Sarthak Ketanbhai Modi and Zi Pong Lim and Yushi Cao and Yupeng Cheng and Yon Shin Teo and Shangwei Lin},
  title = {{SOSAE: Self-Organizing Sparse AutoEncoder}},
  year = {2025},
  eprint = {2507.04644},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04644},
}