Deep learning thrives on gradients. To train a neural network to perform a task, every operation in the pipeline must be differentiable so that errors can be backpropagated from the final loss function to the network's weights.
However, standard media codecs—such as JPEG, H.264, HEVC, and MP3—rely heavily on discrete, non-differentiable operations. Steps like splitting an image into blocks, rounding floating-point coefficients to integers (quantization), and entropy coding (Huffman or Run-Length Encoding) create a "gradient wall." If you try to train a neural network to pre-process an image before it hits a JPEG encoder, or post-process a video after it leaves an H.264 stream, the mathematical gradients stop dead at the encoder boundary.
While researchers have developed fully learned neural compression models to bypass this issue, deploying them requires replacing the global infrastructure of hardware-accelerated decoders. To achieve the benefits of deep learning optimization without breaking compatibility with legacy hardware, a powerful alternative has emerged: differentiable codec proxies.
The Core Mechanism: Bridging the Gradient Wall
A differentiable codec proxy (or surrogate model) acts as a mathematical stand-in for a standard codec during the training phase. Instead of passing data through the actual non-differentiable compression pipeline, the training loop routes data through a proxy that simulates the codec's degradation, quantization effects, and rate-distortion trade-offs using smooth, continuous functions.
Training Phase:
[Input] ──> [Neural Pre-Processor] ──> [Differentiable Proxy] ──> [Loss / Downstream Task]
│
(Gradients flow backward)
Inference Phase:
[Input] ──> [Neural Pre-Processor] ──> [Standard Codec (JPEG/H.264)] ──> [Standard Decoder]
Because the proxy is differentiable, gradients can flow backward from the final loss function (such as perceptual quality or downstream AI task accuracy) all the way to the upstream neural network. Once training is complete, the proxy is discarded, and the real, non-differentiable codec is swapped back in for deployment.
Researchers design these proxies using three primary strategies:
1. Neural Surrogates (Virtual Codecs)
One approach treats the codec as a black box and trains a separate neural network to mimic its behavior. As noted by Zhibo Chen 0001 et al. in their Editorial to the Special Section on Optimized Image/Video Coding Based on Deep Learning, deep neural networks can effectively simulate the behavior of hand-coded compression algorithms with high accuracy.
For instance, Hanbin Son et al. introduced Auxiliary Codec Networks (ACNs) in Enhanced Standard Compatible Image Compression Framework based on Auxiliary Codec Networks to replicate standard codec degradation, allowing for the joint optimization of pre- and post-processing modules. Similarly, Huihui Bai et al. employed a "virtual codec" network in Multiple Description Convolutional Neural Networks for Image Compression to backpropagate gradients through JPEG compression.
Crucially, Jan P. Klopp demonstrated in How To Exploit the Transferability of Learned Image Compression to Conventional Codecs that the optimization directions discovered by a differentiable surrogate model are highly transferable, meaning a surrogate's loss landscape can successfully guide optimization for entirely different conventional codecs.
2. Mathematical and Algorithmic Approximations
Instead of training a black-box neural network, some proxies replace specific non-differentiable operations with smooth mathematical formulations. In Better Compression with Deep Pre-Editing, Hossein Talebi et al. integrated differentiable approximations of quantization and entropy estimation directly into the training loop, allowing a network to learn how to subtly smooth textures before compression.
For video, Aaron Chadha et al. simulated complex pipelines by approximating motion prediction, discrete cosine transforms (DCT), and quantization in Deep Perceptual Preprocessing for Video Coding. To optimize file sizes directly, Jin Young Choi 0002 et al. developed a proxy using multi-layer perceptrons and LSTMs in Task-Aware Quantization Network for JPEG Image Compression to estimate the final code length of quantized coefficients, bypassing non-differentiable entropy coding schemes.
3. Pseudo-Differentiable Routing
A third strategy applies the actual non-differentiable codec during the forward pass to capture true digital distortions, but completely bypasses it during the backward pass. Mayank Kumar Singh et al. leveraged this "pseudo-differentiable" trick in SilentCipher: Deep Audio Watermarking to train audio watermarking models against complex, non-differentiable audio codecs like MP3 or AAC without losing gradient information.
Applications: What Can We Do With a Proxy?
By unlocking gradient flow through standard codecs, proxies enable several novel architectures that enhance legacy compression infrastructure.
Sandwiched Neural Wrappers
A popular architecture is the "sandwich" framework, which wraps a standard codec between neural pre- and post-processors. In Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers, Berivan Isik et al. showed that wrapping HEVC in neural processors trained via a proxy yields massive bitrate reductions while keeping models small enough for practical deployment. Yueyu Hu et al. extended this concept to 3D video compression in One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing, using a differentiable JPEG simulator proxy to anticipate downstream artifacts and ensure compatibility across various hardware-accelerated codecs.
Optimizing for Machine Vision
Standard codecs are designed to discard details that the human eye cannot perceive. However, computer vision models often rely on these exact high-frequency features. By using a differentiable proxy, networks can learn to compress data specifically for downstream AI models rather than human eyes.
In Preprocessing Enhanced Image Compression for Machine Vision, Guo Lu et al. backpropagated losses from downstream vision tasks through a proxy to train a Neural Preprocessing Module (NPP) that filters out features irrelevant to machines. Similarly, Li zhang et al. emulated block splitting and quantization in A Preprocessing Framework for Video Machine Vision under Compression to optimize video for action recognition, while Christoph Reich et al. used a neural proxy in Deep Video Codec Control for Vision Models to dynamically tune H.264 parameters for machine-centric workflows.
Compression-Aware Rescaling and Watermarking
Proxies also allow networks to anticipate information loss in rescaling and watermarking tasks. In Self-Conditioned Probabilistic Learning of Video Rescaling, Yuan Tian et al. utilized control variates theory alongside a surrogate model to stabilize gradient estimates, allowing a rescaling network to optimize for standard industrial compression pipelines.
To prevent compression from breaking the mathematical bijectivity of invertible architectures, Jinhai Yang et al. integrated a differentiable codec simulator in Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling. For active forensics, Yangming Zhou et al. used a differentiable codec simulator in Robust Watermarking for Video Forgery Detection with Improved Imperceptibility and Robustness to train robust watermarking models that survive heavy video compression on social media platforms.
The Catch: The Generalization Gap
While differentiable proxies unlock gradient flow, they remain approximations of highly complex, discrete operations. This introduces a fundamental challenge: the generalization gap.
As noted by Berivan Isik et al., a performance mismatch often occurs when transitioning from a smooth training proxy to the actual discrete codec during inference. Because the proxy cannot perfectly mirror every edge case of a standard codec, the upstream network may accidentally optimize for artifacts unique to the proxy rather than the target codec.
Furthermore, Mengxi Guo et al. pointed out in Video Compression with Arbitrary Rescaling Network that proxies struggle to capture the diverse, proprietary implementations of hardware-accelerated codecs used across different devices. Minimizing this gap—either by refining the fidelity of the proxies or designing more robust training regimes—remains a central focus of research in this field.
Key Foundational Work
Multiple Description Convolutional Neural Networks for Image Compression (2018) — contributed 2 concept elements to this direction.
Enhanced Standard Compatible Image Compression Framework based on Auxiliary Codec Networks (2020) — contributed 1 concept elements to this direction.
Better Compression With Deep Pre-Editing (2020) — contributed 1 concept elements to this direction.