Abstract
Convolutional neural networks (CNNs) in the frequency domain are significant for extending deep learning principles, yet their frequency domain representation is demanding due to complicated training features. This paper proposes an accurate, stable, and efficient Fourier domain training framework based on bin decomposition and non-saturated activation theory. The framework introduces two key components: the Fourier Domain Exponential Linear Unit (FELU) and the Fourier Domain Pyramid Pooling (FPP) layer. FELU alleviates vanishing gradients and aids convergence, while FPP avoids original cropping or warping steps, improving classification accuracy. Using ResNet-50 as a backbone, top-1 and top-5 classification errors were reduced from 28.85 and 9.55 to 18.63 and 4.05, respectively. Speedup ratios reached up to 4.98x at a batch size of 128 on an NVIDIA RTX 2080 GPU. This framework serves as a baseline for establishing efficient Fourier domain CNN training pipelines.
Introduction
Standard CNNs operate in the spatial domain, where convolution operations are computationally expensive, particularly with large kernels. While convolution in the spatial domain is equivalent to point-wise multiplication in the Fourier domain, training a network entirely in the frequency domain presents significant challenges. Historically, non-linear activation functions like ReLU are difficult to represent in the frequency domain without causing vanishing gradients. Furthermore, FFT-based methods often require fixed input sizes, leading to information loss through image warping or cropping.
The FPPE Framework
The proposed FPPE framework (Fourier-domain, Pyramid Pooling, and Exponential linear unit) addresses these limitations by performing all training operations directly in the frequency domain.
Key Innovations:
- Fourier Domain Exponential Linear Unit (FELU): Utilizing a bin decomposition mechanism, FELU separates Fourier coefficients into positive and negative unit functions. This mimics spatial ELU behavior, allowing negative values to push mean activations toward zero, which stabilizes convergence and prevents gradient saturation.
- Fourier Domain Pyramid Pooling (FPP): This layer decomposes feature maps into a fixed number of "Fourier bins" regardless of input resolution. This allows the network to handle variable image sizes without distorting pre-processing steps.
- Two-Pass Training Strategy: The framework employs a dedicated forward pass for feature calculation and a backward pass for gradient propagation, maintaining the entire pipeline within the Fourier domain to maximize throughput.
Experimental Results
The framework was evaluated using ImageNet and the MetaGram-1 dataset (1,200 labeled images across 1,000 categories). When using ResNet-50 (Re-50) as the backbone, the FPPE framework demonstrated superior performance:
- Accuracy: On MetaGram-1, the top-1 classification error dropped significantly from 28.85% to 18.63%. The top-5 error improved from 9.55% to 4.05%.
- Computational Efficiency: At a batch size of 128, the framework achieved a speedup ratio of 4.98x compared to spatial domain training. Unlike other Fourier methods that degrade in deeper networks, FPPE maintained high throughput across varying depths.
- Robustness: The average difference between predicted classification values and ground truth was only 0.21, indicating high goodness-of-fit.
Conclusion
The FPPE framework provides a robust and efficient baseline for training CNNs entirely in the Fourier domain. By replacing expensive spatial convolutions with point-wise multiplications and introducing stable frequency-domain activations and pooling, the framework achieves higher accuracy and significant speedups. Future work will explore extending these Fourier representation strategies to other architectures, such as Recurrent Neural Networks and Transformers.
Summary updated