In standard convolutional neural networks (CNNs), we often think of the Fourier transform as a diagnostic tool or a trick to speed up specific layers. We might jump into the frequency domain to perform a convolution—where it becomes a simple pointwise multiplication—and then immediately jump back to the spatial domain to apply an activation function like ReLU or a pooling operation.
This "domain hopping" is computationally expensive. Each transition requires an Inverse Fast Fourier Transform (IFFT) and a forward FFT, which can quickly become the bottleneck of the architecture. A growing body of research suggests a more radical approach: staying in the frequency domain for the entire duration of the network.
The Efficiency of Pointwise Operations
The primary intuition behind spectral networks is the Convolution Theorem. In the spatial domain, a convolution requires sliding a filter over an image, an $O(N^2)$ operation for an $N \times N$ kernel. In the frequency domain, this same operation is a pointwise multiplication, which is $O(1)$ per element.
Jinhua Lin and Yu Yao demonstrate in A Fourier Domain Training Framework for Convolutional Neural Networks that by keeping the representation in the Fourier domain, we can replace the most taxing parts of a CNN with simple arithmetic. This isn't just for the forward pass; Jack Valmadre and colleagues showed in CFNet that backpropagation can also be performed entirely within the Fourier domain, enabling high-speed tracking at 75 frames per second.
Solving the Nonlinearity Problem
If convolution is easy in the frequency domain, why do we ever leave? The challenge lies in nonlinearities (like ReLU) and pooling. These operations are trivial in the spatial domain but mathematically complex in the spectral domain. Traditionally, researchers would transform the data back to the spatial domain just to apply a nonlinearity.
To solve this, Xiaohan Zhu and Jian Yang developed theoretical frameworks to derive spectral versions of non-linear activations and pooling in Going Deeper in Frequency Convolutional Neural Network. Similarly, Jinhua Lin introduced the Fourier Domain Exponential Linear Unit (FELU) and Fourier Domain Pyramid Pooling (FPP). These components allow the signal to evolve without ever needing to see a pixel.
Preserving Signal Integrity on the Sphere
Beyond speed, spectral processing offers a solution to the "interpolation problem" in non-Euclidean data. When working with spherical data (like global weather maps or 3D molecules), mapping a sphere onto a flat grid introduces distortions and artifacts.
Carlos Esteves and Ameesh Makadia argue in Scaling Spherical CNNs that performing operations like batch normalization and residual connections directly on spectral coefficients (spherical harmonics) is more than just an optimization—it’s a way to maintain mathematical consistency. By avoiding the constant transition between spherical grids and spectral coefficients, the model avoids the interpolation errors that typically degrade the accuracy of deep spherical models.
The Transform Once Paradigm
The ultimate realization of this idea is the "Transform Once" (T1) architecture. In Transform Once: Efficient Operator Learning in Frequency Domain, Michael Poli and Tri Dao propose a model that performs exactly one transform at the input and one inverse transform at the output.
All hidden layers—the "meat" of the network—operate on spectral coefficients. This approach achieves up to 10x speedups because it eliminates the $O(L \cdot N \log N)$ cost of transforming at every one of the $L$ layers.
Choosing the Right Transform
While the Fourier transform is the most common, it involves complex numbers, which can be cumbersome for hardware. Jianwei Ma explored the use of the Hartley Transform in Hartley Spectral Pooling for Deep Learning. Because the Hartley transform is real-valued, it preserves the structural information of the image while keeping the arithmetic simple and real, avoiding the memory overhead of complex-valued representations.
By treating the frequency domain as the primary workspace rather than a temporary shortcut, these researchers are building a new class of "all-spectral" networks that are faster, more mathematically robust, and better suited for high-resolution and non-Euclidean data.