Modern image and video compression standards like JPEG, H.264, and HEVC are the bedrock of digital communication. They are highly optimized and hardware-accelerated, but they share a fundamental flaw for deep learning: they are non-differentiable. Because these codecs rely on discrete operations like integer transforms and quantization (rounding), gradients cannot flow through them. This "black box" nature prevents researchers from using standard backpropagation to train neural networks that work in tandem with existing infrastructure.
To bridge this gap, a growing body of research focuses on differentiable proxies—neural or mathematical surrogates that mimic the behavior of standard codecs while providing a continuous gradient path for optimization.
The Non-Differentiable Bottleneck
In a typical neural image processing pipeline, we might want to train a pre-processor to "clean" an image before it is compressed, or a post-processor to remove artifacts after it is decoded. However, as noted by Muhammad Umar Karim Khan and Amir Said, the hard-coded logic and discrete quantization steps of standard codecs act as a barrier. If you cannot calculate the derivative of the compression step, you cannot tell the pre-processor how to change its weights to improve the final output.
The Virtual Codec as a Gradient Bridge
The core solution is the Virtual Codec Network (VCN) or surrogate model. Instead of training against the actual JPEG or H.264 engine, researchers train a neural network to simulate the engine. Huihui Bai and Lijun Zhao pioneered this by using a "Virtual Codec" to approximate the degradation effects of standard compression (Learning a Virtual Codec Based on Deep Convolutional Neural Network to Compress Image).
During training, the system looks like this:
- Forward Pass: The input passes through a neural pre-processor, then through the actual non-differentiable codec (or a high-fidelity proxy).
- Backward Pass: Gradients are calculated from the final loss (e.g., image quality or AI task accuracy). When the gradient reaches the codec, it passes through the differentiable surrogate instead of the real codec.
This allows the pre-processor to "see" through the compression bottleneck. Jan P. Klopp demonstrated that optimization directions learned from these surrogates are remarkably transferable to the real, non-differentiable codecs they represent (How To Exploit the Transferability of Learned Image Compression to Conventional Codecs).
Sandwiched Compression and Neural Wrapping
This mechanism enables a framework often called Sandwiched Compression or Neural Wrapping. As described by Onur G. Guleryuz and Yueyu Hu, a standard codec is "sandwiched" between a neural pre-processor and post-processor (Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers).
The intuition is that the pre-processor learns to "pre-distort" the image in a way that makes it more resilient to the specific artifacts—like blocking or ringing—that the codec will eventually introduce. For example, Jinhai Yang showed that by using a differentiable codec simulator, a network can learn to rescale images such that they retain high-frequency details even after being crushed by JPEG quantization (Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling).
Beyond Human Vision: Optimizing for Machines
Standard codecs are designed to look good to humans, often discarding data that is "invisible" to us but critical for AI. Christoph Reich and Srimat Chakradhar used differentiable proxies to solve this by training a control network that tunes H.264 parameters specifically for downstream vision tasks like object detection (Deep Video Codec Control for Vision Models). By propagating gradients from the detection loss back through a surrogate codec, the system learns which compression settings preserve the features most important to the AI, rather than the human eye.
Advancing Proxy Fidelity
The effectiveness of this approach depends on how well the proxy matches the real codec. Early work used simple approximations like the Straight-Through Estimator (STE). More recent work has pushed for higher fidelity:
- Mathematical Refinement: Christoph Reich developed a "Differentiable JPEG" that models subtle details like floor functions and clipping, which are often ignored but critical at high compression rates (Differentiable JPEG: The Devil is in the Details).
- Generative Surrogates: Mingwei He and Tongda Xu proposed using pre-trained diffusion models as differentiable proxies, treating compression as a conditional generation task to better anticipate complex artifacts (Versatile Recompression-Aware Perceptual Image Super-Resolution).
- Hybrid Training: Some researchers, like Esteban Pesnel, argue for using the actual non-differentiable codec in the forward pass and only using the surrogate for the gradient in the backward pass, ensuring the model experiences real-world artifacts during training (SCALED: Surrogate-gradient for Codec-Aware Learning of Downsampling in ABR Streaming).
By treating legacy codecs as differentiable components, this research direction allows the "old" world of standard-compliant hardware to benefit from the "new" world of end-to-end deep learning.