Frequency Factorization for Efficient Neural Networks

This direction is split into other directions: octave convolution. Its last version stays readable and citable here.

Version 1 of 141 papers64 conceptsfrozen 2026-10-04

In standard convolutional neural networks, every pixel is treated with equal importance. Whether a region contains a sharp edge or a smooth, blurry sky, the network applies the same high-resolution filters across the entire feature map. This approach is computationally expensive and ignores a fundamental property of natural images: spatial redundancy.

Most information in an image is "low-frequency"—it changes slowly across space. By treating these smooth signals with the same density as sharp textures, traditional architectures waste significant memory and compute. A growing body of research, centered around the concept of Frequency Factorization, seeks to solve this by decomposing feature maps into high- and low-frequency components and processing them at different scales.

The Redundancy Problem

In a typical CNN, feature maps are spatially redundant. If you look at the activations in a hidden layer, neighboring values are often highly correlated. Haoqi Fan and colleagues argue in Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution that this redundancy is a byproduct of failing to distinguish between different types of information.

Global structures (low frequency) describe the "what" and "where" of an object, while fine details (high frequency) describe the "how" (textures and edges). Standard layers process both at the same resolution, which is akin to using a microscope to look at a mountain.

Octave Convolutions and Spatial Splitting

The most influential mechanism for addressing this is the Octave Convolution (OctConv). The intuition is simple: if the low-frequency component is spatially smooth, we can safely store and process it at a lower resolution—specifically, one "octave" (a factor of two) lower.

As detailed by Haoqi Fan, an OctConv layer splits the input channels into two groups:

  1. High-frequency path: Processes fine details at the original resolution.
  2. Low-frequency path: Processes global structures at half the resolution.

Crucially, these paths are not isolated. The architecture includes "cross-frequency" communication, where information from the low-resolution path is upsampled to guide the high-resolution path, and vice versa. This allows the model to maintain a large receptive field (via the low-res path) without losing the ability to localize fine features.

This principle has been adapted across various domains. For instance, Sangjune Shin introduced Octave-YOLO, which slashes the computational cost of real-time object detection by nearly half while maintaining accuracy. Similarly, Dayvid Castro utilized this decomposition in DANCER to separate character-level details from page-level layout signals in handwriting recognition.

Why Frequency Decomposition Works

The success of this approach stems from three primary advantages:

  • Computational Efficiency: By processing the low-frequency branch at a reduced resolution, the number of floating-point operations (FLOPs) drops significantly. Jinmin Li and others demonstrated in FSR that high-frequency components actually require less neural capacity than the base structures for effective reconstruction, allowing for further channel reductions.
  • Expanded Receptive Field: A convolution kernel of size $3 \times 3$ on a half-resolution map effectively covers the same area as a $6 \times 6$ kernel on a full-resolution map. This allows the network to capture global context more efficiently, a benefit highlighted in medical imaging tasks like xQSM and AbdomenNet.
  • Feature Disentanglement: Separating frequencies acts as a form of regularization. In Inception Transformer (iFormer), Chenyang Si and colleagues used frequency splitting to prevent Vision Transformers from acting as simple low-pass filters, ensuring that local details like edges are preserved alongside global attention.

Generalizing the Frequency Perspective

The idea of frequency-aware processing extends beyond simple resolution downsampling. Researchers have explored more explicit mathematical transforms to achieve similar ends:

  • Wavelets: Tianshui Chen et al. proposed a Wavelet-like Auto-Encoder that forces high-frequency components to be sparse, concentrating discriminative information into a low-resolution core. More recently, Haodong Zhu introduced WaveMamba, which uses wavelet decomposition to fuse RGB and Infrared data more efficiently than attention-heavy models.
  • Discrete Cosine Transform (DCT): In DCT-SNN, Isha Garg used DCT to feed the most important frequency components to a Spiking Neural Network first, drastically reducing inference latency.
  • Caching and Diffusion: The concept is even being used to accelerate generative models. Jiacheng Liu developed FreqCa, which accelerates Diffusion Transformers by decoupling structural components from fine details and caching them independently, reducing memory usage by 99%.

Impact on Robustness and Real-World Data

Beyond speed, frequency factorization improves how models handle "messy" real-world data. Sabine Müller found that frequency-based decomposition makes brain tumor segmentation more robust to noise by encouraging the model to prioritize global context over local artifacts. In aerial imaging, Chengyuan Li used frequency-divided feature maps in OcSaFPN to detect objects in noisy environments without needing a separate denoising step.

By recognizing that an image is a composite of signals with different roles and scales, these researchers have moved away from the "one-size-fits-all" approach of standard convolutions. The result is a family of architectures that are not only faster and lighter but also more aligned with the underlying structure of the visual world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_frequency_factorization_for_efficient_ne_5aa8d8d4,
  title = {Frequency Factorization for Efficient Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/frequency-factorization-for-efficient-neural-networks/rdr_5aa8d8d41f8740a8800650602d14f047},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_frequency_factorization_for_efficient_ne_5aa8d8d4_v1,
  title = {Frequency Factorization for Efficient Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/frequency-factorization-for-efficient-neural-networks/rdr_5aa8d8d41f8740a8800650602d14f047/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}