Abstract
Autoencoders (AE) are essential in learning representation of large data (like images) for dimensionality reduction. Images are converted to sparse domain using transforms like Fast Fourier Transform (FFT) or Discrete Cosine Transform (DCT) where information that requires encoding is minimal. By optimally selecting the feature-rich frequencies, we are able to learn the latent vectors more robustly. We successfully show enhanced performance of autoencoders in sparse domain for images.
Introduction
Autoencoders (AEs) are a staple of unsupervised learning, designed to compress data into a latent bottleneck and then reconstruct it. However, they face two persistent hurdles: memorization versus learning, and architectural dependency. In the spatial domain (raw pixels), information is spread out, making every pixel appear equally important even though much of the data is redundant. This often leads to models memorizing individual data points rather than learning underlying distributions. Furthermore, standard AE performance is heavily tied to decoder complexity; shallow decoders often struggle to reconstruct fine spatial details.
This paper proposes transforming images into a sparse frequency domain (using FFT or DCT) before training. By changing the "language" the autoencoder speaks, we can make the learning process more efficient and robust.
Methods
The proposed approach involves a "Sparse Domain" transformation prior to neural network processing.
- Transformations: Data undergoes Fast Fourier Transform (FFT) or Discrete Cosine Transform (DCT) to convert images from the spatial domain to the frequency domain.
- Sparsity and Feature Selection: In the frequency domain, structural information (edges and shapes) is concentrated in low-frequency components. By focusing on these feature-rich frequencies, the model avoids initial struggles with high-frequency noise.
- Architecture and Dataset: The study utilizes the MNIST dataset (60,000 training, 10,000 test images), each $28 \times 28$ pixels. The baseline architecture is a two-layered linear autoencoder compressing 784 dimensions to a latent vector of 20.
Results and Discussion
The primary finding is that the frequency-domain model (AE-FFT) learns significantly faster and more effectively than its spatial-domain counterpart.
Rapid Convergence and Lower Loss
One of the most striking observations is the speed of convergence. While vanilla autoencoders require several epochs to stabilize, the FFT-based model reaches near-optimal loss values after just the first epoch. As shown in Figure 1, AE-FFT consistently maintains a lower binary cross-entropy loss throughout the training process.
Robustness to Architectural Simplification
Typically, reducing the number of layers in a decoder leads to a significant drop in performance. However, the frequency domain makes the reconstruction task fundamentally easier. Experiments revealed that a 1-layered AE-FFT actually outperforms a standard 2-layered vanilla AE. This suggests that the sparse representation allows for high-quality reconstruction even with significantly reduced computational complexity.
Conclusion
Training autoencoders in the sparse frequency domain offers a robust alternative to traditional spatial-domain training. It enables faster convergence and allows for simpler decoder architectures without sacrificing reconstruction quality. Future work should explore the scalability of this approach to more complex datasets like ImageNet, develop rigorous frameworks for frequency selection, and investigate the integration of phase information to better preserve edge details.
Summary updated