RGB No More: Minimally-Decoded JPEG Vision Transformers

Jeongsoo Park, Justin JohnsonCVPR 2023 · DOI · arXiv

Abstract

Modern computer vision pipelines typically decode compressed JPEG images into raw RGB pixels before processing them with neural networks. This process is computationally expensive and often creates a bottleneck at the CPU level. We present a "minimally-decoded" pipeline that trains Vision Transformers (ViTs) directly on JPEG Discrete Cosine Transform (DCT) coefficients. By aligning the block-based nature of JPEG compression with the patch-based architecture of ViTs and implementing native frequency-domain data augmentations, we achieve up to a 39.2% speedup in training and 17.9% faster inference without sacrificing accuracy.

Introduction: The Decoding Bottleneck

Almost every digital image used in deep learning starts as a compressed JPEG file. Standard workflows follow a rigid "decode-first" approach: images are loaded from disk, decoded into an RGB pixel grid, and then fed into a model. This routine has two primary drawbacks. First, decoding is a CPU-intensive task that often leaves high-performance GPUs idle while waiting for data. Second, modern Vision Transformers (ViTs) operate by partitioning images into patches—a process that is redundant because JPEG compression already organizes data into $8 \times 8$ frequency-domain blocks called DCT coefficients.

Comparison of the typical RGB pipeline versus the proposed minimally-decoded DCT pipeline, showing the shift from CPU-heavy decoding to direct GPU processing.

Methodology: Training in the Frequency Domain

The proposed approach skips full JPEG decoding, performing only the initial entropy decoding to extract DCT coefficients. This bypasses the most expensive stages of the JPEG pipeline, such as inverse DCT and color space conversion.

Modified Patch Embedding

To integrate $8 \times 8$ DCT blocks into the ViT architecture, we designed specialized embedding strategies to handle the different resolutions of luma (brightness) and chroma (color) data:

  • Grouped Embedding: Spatially coincident DCT blocks are grouped and embedded together.
  • Separate Embedding: Individual blocks are embedded separately before being mixed in subsequent layers.
  • Concatenated Embedding: Luma and chroma channels are processed through distinct pathways and then joined.

Furthermore, we utilize sub-block conversion, a mathematical technique that allows us to transform $8 \times 8$ blocks into different patch sizes directly within the frequency domain, maintaining architectural flexibility.

DCT-Domain Data Augmentation

Effective training of ViTs requires robust data augmentation. Traditionally, this is done in the RGB domain. We implemented a suite of "DCT-native" augmentations that operate directly on frequency coefficients:

  • Photometric Augmentation: We adjust the "DC" coefficient (representing average block brightness) to modify contrast and brightness. Scaling "AC" coefficients allows for sharpness adjustments.
  • Geometric Augmentation: Flips and 90-degree rotations are achieved by reordering or transposing the coefficient matrices, avoiding the need for spatial interpolation.
Overview of photometric and geometric data augmentations implemented directly on DCT coefficients, bypassing the need for RGB conversion.

Experimental Results

We evaluated our "JPEG-ViT" models on the ImageNet-1k dataset, comparing them against standard RGB-trained ViT-Ti and ViT-S baselines.

  • Efficiency Gains: The JPEG-ViT-Ti model demonstrated a 39.2% increase in training speed and a 17.9% improvement in inference throughput. By reducing the data loading workload by up to 87.6%, we effectively eliminated the CPU bottleneck.
  • Accuracy Preservation: There was no loss in top-1 accuracy. In several instances, the DCT-trained models outperformed their RGB counterparts by approximately 1.0%, suggesting that frequency-domain features may provide a beneficial inductive bias for Transformers.

Discussion and Conclusion

The core insight of this work is the fundamental compatibility between the block-based structure of JPEG and the patch-based nature of Vision Transformers. By modifying only the initial patch embedding layer, the rest of the Transformer architecture remains standard, allowing it to "think" in the same compressed language used for storage.

While highly effective, some limitations remain. Complex geometric transformations like arbitrary rotations or shearing are mathematically difficult to perform directly on DCT coefficients. Additionally, while JPEG is ubiquitous, future research is needed to adapt these techniques to newer formats like HEIF or AVIF. Nevertheless, this work demonstrates that recovering RGB pixels is often unnecessary, paving the way for more efficient, end-to-end compressed-domain vision systems.

Summary updated

BibTeX
@inproceedings{park2023rgb,
  author = {Jeongsoo Park and Justin Johnson},
  title = {{RGB No More: Minimally-Decoded JPEG Vision Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02139},
}
Figure