Scaling Neural Networks with Frequency Domain Convolutions

This direction is merged into another direction: frequency-domain convolution. Its last version stays readable and citable here.

Version 1 of 193 papers105 conceptsfrozen 2026-10-04

The fundamental operation of modern computer vision—the convolution—is computationally expensive. As we move toward higher-resolution images and larger receptive fields, the standard "sliding window" approach in the spatial domain begins to hit a wall. To overcome this, a growing body of research suggests a mathematical detour: shifting operations into the frequency domain.

By leveraging the Convolution Theorem, researchers are replacing heavy spatial convolutions with efficient element-wise multiplications, enabling models to process data faster and capture global patterns that were previously out of reach.

The Spectral Shortcut

In the spatial domain, convolving an image with a filter requires a nested loop of multiplications for every pixel and every kernel position. This results in a computational complexity that scales quadratically with the size of the input and the kernel.

The frequency domain offers a shortcut. The Convolution Theorem states that a convolution in the spatial domain is equivalent to a point-wise multiplication in the frequency domain. By applying a Fast Fourier Transform (FFT) to both the input and the filter, we can perform the "convolution" as a simple Hadamard product. As Michael Mathieu, Mikael Henaff, and Yann LeCun demonstrated in Fast Training of Convolutional Networks through FFTs, this shift reduces complexity from $O(N^2 K^2)$ to $O(N^2 \log N)$, where $N$ is the image dimension.

This efficiency is particularly transformative for large kernels. While traditional CNNs often stick to $3 \times 3$ or $5 \times 5$ filters to save on computation, frequency-domain methods allow for filters as large as the image itself with minimal extra cost. Nicolas Vasilache and Jeff Johnson showed that the speedup of these spectral methods increases significantly as the spatial size of the kernel grows, making them ideal for tasks requiring broad context.

Achieving a Global Receptive Field

Beyond speed, the frequency domain changes what a network can see. In a standard CNN, the "receptive field" (the area of the input that influences a specific output neuron) grows slowly, layer by layer. In the frequency domain, every spectral coefficient contains information from the entire spatial signal.

This "global" view is essential for fields like 3D microscopy and MRI reconstruction. Diptodip Deb and Alex Bo-Yuan Chen utilized this in FourierNets to decode non-local optical signals, while Xiaohan Liu and Yanwei Pang developed Faster Fourier Convolution to accelerate MRI scans by capturing global dependencies in a single pass.

Even when given the freedom of "infinitely large" filters, the network can learn to be selective. Janis Keuper and Margret Keuper explored this in Neural Implicit Frequency Filters, finding that while spectral methods allow for global kernels, models often naturally optimize toward well-localized filters that are more effective for feature extraction.

Real-Time Processing for Sparse and Temporal Data

The efficiency of spectral operations has made them the "gold standard" for real-time applications on constrained hardware.

  1. Visual Tracking: Jorge Batista 0001 pioneered the use of circulant matrices in the Fourier domain for Kernelized Correlation Filters. By diagonalizing these matrices in the frequency domain, complex tracking tasks are reduced to simple divisions, allowing trackers to run at over 170 FPS.
  2. Event Cameras: Data from event cameras is sparse and asynchronous. Hongwei Ren and Fei Ma developed FECNet, which maps these "event clouds" into the frequency domain. This allows for high-speed action recognition and pose estimation by bypassing the need for expensive spatial convolutions on high-resolution streams.
  3. Time Series Analysis: Modeling long-term periodicities in time series is difficult with sliding windows. Kun Yi and Qi Zhang have shown that Frequency Transformation allows models to capture seasonal trends and long-range dependencies more naturally and with significantly less computational overhead than time-domain methods.

The Cost of the Shortcut

While the frequency domain offers massive speedups, it is not a free lunch. The FFT assumes periodic boundary conditions—essentially treating the image as if it wraps around like a torus. If not handled carefully, this can introduce artifacts at the edges of the data. Furthermore, for very small kernels (like $1 \times 1$ convolutions), the overhead of performing the FFT and its inverse can actually be slower than the spatial operation, as noted by Neil Adit in Dense Pruning of Pointwise Convolutions.

The research direction represented by these papers suggests a hybrid future: using spatial convolutions for local, fine-grained details and spectral operations for global context and high-speed scaling. By moving between domains, we can build architectures that are both mathematically elegant and computationally lean.

Papers in this version

Showing 60 of 93 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_neural_networks_with_frequency_d_811dbfed,
  title = {Scaling Neural Networks with Frequency Domain Convolutions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-neural-networks-with-frequency-domain-convolutions/rdr_811dbfed06084a679a7b72794e891596},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_neural_networks_with_frequency_d_811dbfed_v1,
  title = {Scaling Neural Networks with Frequency Domain Convolutions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-neural-networks-with-frequency-domain-convolutions/rdr_811dbfed06084a679a7b72794e891596/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}