A Fourier Domain Training Framework for Convolutional Neural Networks Based on the Fourier Domain Pyramid Pooling Method and Fourier Domain Exponential Linear Unit

Jinhua Lin, Lin Ma, Yu YaoIEEE Access 2019 · DOI

Abstract

Convolutional neural networks (CNNs) in the frequency domain are significant for extending deep learning principles, yet their frequency domain representation is demanding due to complicated training features. This paper proposes an accurate, stable, and efficient Fourier domain training framework based on bin decomposition and non-saturated activation theory. The framework introduces two key components: the Fourier Domain Exponential Linear Unit (FELU) and the Fourier Domain Pyramid Pooling (FPP) layer. FELU alleviates vanishing gradients and aids convergence, while FPP avoids original cropping or warping steps, improving classification accuracy. Using ResNet-50 as a backbone, top-1 and top-5 classification errors were reduced from 28.85 and 9.55 to 18.63 and 4.05, respectively. Speedup ratios reached up to 4.98x at a batch size of 128 on an NVIDIA RTX 2080 GPU. This framework serves as a baseline for establishing efficient Fourier domain CNN training pipelines.

Introduction

Standard CNNs operate in the spatial domain, where convolution operations are computationally expensive, particularly with large kernels. While convolution in the spatial domain is equivalent to point-wise multiplication in the Fourier domain, training a network entirely in the frequency domain presents significant challenges. Historically, non-linear activation functions like ReLU are difficult to represent in the frequency domain without causing vanishing gradients. Furthermore, FFT-based methods often require fixed input sizes, leading to information loss through image warping or cropping.

The FPPE Framework

The proposed FPPE framework (Fourier-domain, Pyramid Pooling, and Exponential linear unit) addresses these limitations by performing all training operations directly in the frequency domain.

Comparison of CNN, fbFFT, and the proposed FPPE architectures with corresponding accuracy plots.

Key Innovations:

  • Fourier Domain Exponential Linear Unit (FELU): Utilizing a bin decomposition mechanism, FELU separates Fourier coefficients into positive and negative unit functions. This mimics spatial ELU behavior, allowing negative values to push mean activations toward zero, which stabilizes convergence and prevents gradient saturation.
  • Fourier Domain Pyramid Pooling (FPP): This layer decomposes feature maps into a fixed number of "Fourier bins" regardless of input resolution. This allows the network to handle variable image sizes without distorting pre-processing steps.
  • Two-Pass Training Strategy: The framework employs a dedicated forward pass for feature calculation and a backward pass for gradient propagation, maintaining the entire pipeline within the Fourier domain to maximize throughput.
Detailed schematic of the forward (Ffpass) and backward (Fbpass-1, Fbpass-2) training passes in the Fourier domain.

Experimental Results

The framework was evaluated using ImageNet and the MetaGram-1 dataset (1,200 labeled images across 1,000 categories). When using ResNet-50 (Re-50) as the backbone, the FPPE framework demonstrated superior performance:

  • Accuracy: On MetaGram-1, the top-1 classification error dropped significantly from 28.85% to 18.63%. The top-5 error improved from 9.55% to 4.05%.
  • Computational Efficiency: At a batch size of 128, the framework achieved a speedup ratio of 4.98x compared to spatial domain training. Unlike other Fourier methods that degrade in deeper networks, FPPE maintained high throughput across varying depths.
  • Robustness: The average difference between predicted classification values and ground truth was only 0.21, indicating high goodness-of-fit.

Conclusion

The FPPE framework provides a robust and efficient baseline for training CNNs entirely in the Fourier domain. By replacing expensive spatial convolutions with point-wise multiplications and introducing stable frequency-domain activations and pooling, the framework achieves higher accuracy and significant speedups. Future work will explore extending these Fourier representation strategies to other architectures, such as Recurrent Neural Networks and Transformers.

Summary updated

Abstract

convolutional neural networks (CNNs) in the frequency domain is of great significance for extending the deep learning principle to the frequency domain. However, the frequency domain representation of the convnet architecture is highly demanding due to their complicated Fourier domain training features. Therefore, high accuracy and unambiguous representation strategies are needed for training convolutional neural networks entirely in the Fourier domain. Being founded on the bin decomposition mechanism and the non-saturated activation theory, this paper proposes an accurate, stable and efficient Fourier domain training framework for convolutional neural networks. The framework contains two important Fourier domain representations: one is the Fourier domain exponential linear unit, and the other is the pyramid pooling layer. The former alleviates the vanishing phenomenon and makes CNNs easier to converge in the Fourier domain; the latter avoids the original cropping or warping steps and improves the classification accuracy. With the framework, the Fourier domain training accuracy is improved without sacrificing the throughput of the graphic processing unit (GPU). With the Re-50 as the backbone, the top-1 and top-5 classification errors are reduced from 28.85 and 9.55 to 18.63 and 4.05, respectively, while the speedup ratios of the framework can reach up to 4.9877 and 1.8997, respectively, at a batch size of 128 on an NVIDIA GEFORCE RTX 2080 GPU (8.92 TFLOPS). The average difference between the classification value and the ground truth value is only 0.21 on the MetaGram-1 set, which indicates great goodness-of-fit and robustness of the framework. This investigation illustrates that the proposed Fourier domain CNN framework using the sophisticated Fourier domain representation strategy is highly efficient and accurate. Therefore, it may serve as a baseline framework to establish the training pipelines for Fourier domain CNNs, which can improve the deep learning accuracy of CNNs and extend the Fourier domain representation strategy to other deep learning networks.
BibTeX
@inproceedings{lin2019fourier,
  author = {Jinhua Lin and Lin Ma and Yu Yao},
  title = {{A Fourier Domain Training Framework for Convolutional Neural Networks Based on the Fourier Domain Pyramid Pooling Method and Fourier Domain Exponential Linear Unit}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2936591},
}
Figure