Computer Vision Directly on Compressed Frequency Coefficients

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 124 papers28 conceptsfrozen 2026-10-04

The standard pipeline for computer vision is surprisingly redundant. Almost every image used to train a model is stored in a compressed format like JPEG. To process these images, we traditionally decompress them into raw RGB pixels, only for the first few layers of a neural network to immediately perform convolutions that act as frequency filters—essentially re-extracting the information that was already present in the compressed bitstream.

A growing body of research suggests we should skip this "pixel bottleneck" entirely. By training models directly on the frequency-domain coefficients used in compression, researchers are achieving massive speedups and improved robustness without sacrificing accuracy.

The Pixel Bottleneck

In a typical deep learning workflow, the CPU spends significant cycles decoding JPEGs into large RGB tensors, which are then shuffled to the GPU. Lionel Gueguen and colleagues in Faster Neural Networks Straight from JPEG pointed out that this decompression is not just a computational tax; it is a loss of structured information.

JPEG compression works by breaking an image into $8 \times 8$ blocks and applying a Discrete Cosine Transform (DCT). This transform converts spatial pixels into frequency coefficients: a "DC" term representing the average brightness and several "AC" terms representing textures and edges at different scales. When we decompress, we throw away this mathematical decomposition, only to have the neural network try to relearn it through early convolutional layers.

From Pixels to Frequency Channels

The core intuition behind "Compressed Domain Vision" is to treat these DCT coefficients as input features. Instead of a high-resolution RGB image, the network sees a smaller spatial grid where each "pixel" is actually a vector of frequency coefficients.

For example, a $224 \times 224$ RGB image can be viewed as a $28 \times 28$ grid of $8 \times 8$ blocks. In the frequency domain, this becomes a $28 \times 28$ input with 64 channels (one for each DCT coefficient) per color component. Kai Xu and Fengbo Ren demonstrate in Learning in the Frequency Domain that this representation allows the model to preserve structural information that is often lost during standard spatial downsampling.

This approach is particularly natural for Vision Transformers (ViTs). Jeongsoo Park and Justin Johnson show in RGB No More: Minimally-Decoded JPEG Vision Transformers that the block-based nature of JPEG aligns perfectly with the patch-based processing of ViTs. By feeding DCT blocks directly into the transformer, they achieved a 39.2% training speedup and 17.9% faster inference.

Efficiency: Skipping the Decompression Tax

The most immediate benefit of this direction is raw speed. By operating in the compressed domain, models can:

  1. Eliminate Decoding: Skip the CPU-heavy JPEG decompression step.
  2. Reduce Bandwidth: Transfer smaller frequency-domain tensors from CPU to GPU.
  3. Lower Resolution Requirements: Hao Feng and Hao Liu show in DocPedia that frequency-domain processing allows Large Multimodal Models to "see" high-resolution documents (up to $2560 \times 2560$) using significantly fewer visual tokens than pixel-domain models.

Kfir Goldberg further validated this with the eFUN architecture in Rethinking FUN: Frequency-Domain Utilization Networks, which was 65% faster than EfficientNet-B0 by leveraging these pre-digested features.

Robustness and Resolution: The Hidden Benefits

Beyond speed, frequency-domain training offers unique architectural advantages.

Adversarial Robustness: Standard CNNs are often fooled by high-frequency noise that is invisible to humans. Chin-Yuan Yeh and Ming-Syan Chen found that replacing early layers with a fixed Spatial Frequency (SF) layer—which mimics the Human Visual System's preference for low frequencies—significantly boosts resistance to adversarial attacks. Their work, Evaluating Adversarial Robustness in the Spatial Frequency Domain, suggests that the frequency domain is a more "natural" space for vision models to operate.

Storage Efficiency: Minhyun Lee and Dongyoon Han demonstrated in SeiT++ that models can be trained using just 1% of the storage required for traditional pixel-based datasets by training directly on compressed coefficients. This makes high-performance training viable even in resource-constrained environments.

Closing the Perception Gap

For decades, image codecs like JPEG were designed to please the human eye by discarding information humans don't notice. However, Christoph Reich and Stefan Roth warn in A Perspective on Deep Vision Performance with Standard Image and Video Codecs that this "human-centric" compression can be detrimental to AI.

By moving the model's "eyes" directly to the bitstream, we allow the network to decide which frequencies are important for semantic understanding, rather than relying on a decompression process that might introduce artifacts or lose critical data. As vision models move toward edge devices and real-time processing, the shift from pixel-centric to frequency-centric architectures represents a fundamental optimization of the entire visual intelligence pipeline.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computer_vision_directly_on_compressed_f_5b081d20,
  title = {Computer Vision Directly on Compressed Frequency Coefficients},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computer-vision-directly-on-compressed-frequency-coefficients/rdr_5b081d20ead3428e85e07d8c402309fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_computer_vision_directly_on_compressed_f_5b081d20_v1,
  title = {Computer Vision Directly on Compressed Frequency Coefficients},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computer-vision-directly-on-compressed-frequency-coefficients/rdr_5b081d20ead3428e85e07d8c402309fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}