To train a neural network using gradient descent, every operation in the pipeline must have a well-defined derivative. This requirement creates a fundamental conflict when deep learning meets data compression.
Compression inherently requires mapping continuous representations (like network weights or latent feature maps) into discrete tokens (like bits or integers) so they can be stored or transmitted. This mapping is achieved via quantization, such as a rounding function $\lfloor x \rceil$ or a hard vector lookup.
The mathematical problem with quantization is simple but devastating for backpropagation: a rounding function is a step function. Its derivative is exactly zero everywhere except at the integers, where it is completely undefined.
$$\frac{\partial \lfloor x \rceil}{\partial x} = 0 \quad \text{for all } x \notin \mathbb{Z}$$
If you attempt to pass a gradient backward through a quantizer, it is multiplied by zero. As a result, the layers preceding the quantizer—such as an encoder or a feature extractor—receive no gradient signal and cannot learn. This non-differentiability acts as an optimization bottleneck across many domains, including image compression (Lucas Theis et al. in Lossy Image Compression with Compressive Autoencoders), video compression (Guo Lu et al. in DVC: An End-To-End Deep Video Compression Framework), and digital transmission of wireless channel state information (Zhengyang Hu et al. in A Learnable Optimization and Regularization Approach to Massive MIMO CSI Feedback).
To bypass this bottleneck, researchers use differentiable proxies—continuous relaxations that simulate the effects of quantization during training while allowing gradients to flow unimpeded.
Continuous Relaxation via Additive Uniform Noise
The foundational strategy for training neural compressors is to replace deterministic rounding with additive noise during the forward pass of training.
In their seminal work on nonlinear transform coding, Jona Ballé et al. (End-to-end optimization of nonlinear transform codes for perceptual quality) proposed adding uniform noise $\mathcal{U}(-0.5, 0.5)$ to the continuous latent representations instead of rounding them:
$$x_{\text{train}} = x + \Delta, \quad \Delta \sim \mathcal{U}(-0.5, 0.5)$$
During inference, the noise is dropped, and standard rounding is restored: $x_{\text{inference}} = \lfloor x \rceil$.
Why It Works
Adding uniform noise acts as a continuous proxy for two reasons:
- Gradient Flow: The derivative of the addition operation with respect to the input is exactly 1 ($\frac{\partial (x + \Delta)}{\partial x} = 1$). Gradients pass backward through the layer completely unaltered.
- Distributional Matching: The probability density function of the noise-corrupted representation closely matches the continuous envelope of the discrete histogram produced by actual rounding. This allows the network's entropy estimator to accurately model and minimize the expected bitrate during training, as further detailed by Jona Ballé et al. in Nonlinear Transform Coding.
Soft Assignments and Soft Bits
When compression requires mapping continuous vectors to discrete items in a codebook (Vector Quantization) or allocating specific bit depths, the bottleneck manifests as a hard argmin or discrete selection step. The solution here is to relax the hard choice into a soft, weighted average.
For instance, in retrieval systems where continuous image features must be compressed into discrete codewords, Junsong Yuan et al. (Product Quantization Network for Fast Image Retrieval) replaced the non-differentiable hard assignment with a soft, softmax-based distribution over the codebook. Instead of selecting the single closest codeword, the network computes a soft representation based on distances to all codewords. As training progresses, a temperature parameter is annealed to force the soft distribution to converge toward a hard assignment.
Similarly, the authors of Learned Image Compression with Soft Bit-based Rate-Distortion Optimization bypassed the zero-gradient problem of arithmetic coding by approximating discrete bit allocations with differentiable sigmoid functions. This allowed the network to directly optimize the rate-distortion trade-off by treating bit allocation as a continuous probability.
Differentiable Surrogates for Legacy Codecs
A distinct challenge arises when deep networks must interact with legacy, hand-engineered compression standards like JPEG or BPG. These codecs are hard-coded and completely non-differentiable, preventing researchers from optimizing neural networks to produce images that are inherently easier for standard codecs to compress.
To solve this, researchers construct a differentiable surrogate—a neural network or mathematical proxy that mimics the behavior of the legacy codec during training.
Training Path:
[Input Image] ──> [Neural Pre-processor] ──> [Differentiable Codec Proxy] ──> [Downstream Task / Loss]
│ (Gradients flow back)
Inference Path: ▼
[Input Image] ──> [Neural Pre-processor] ──> [Standard Legacy Codec] ──> [Standard Decoder]
Jan P. Klopp et al. leveraged this concept in How To Exploit the Transferability of Learned Image Compression to Conventional Codecs. They trained a neural pre-processing filter by passing images through a differentiable neural codec proxy. Because the proxy provides a valid gradient path, the pre-processor successfully learns how to "pre-edit" images to maximize compression efficiency before they are handed off to standard, unmodified hardware decoders.
This surrogate approach is highly effective for task-specific compression:
- Optimizing for Machine Vision: Guo Lu et al. (Preprocessing Enhanced Image Compression for Machine Vision) used a neural preprocessing module to filter out semantic details irrelevant to AI models before BPG compression.
- Optimizing Codec Parameters: Xiyang Luo et al. (The Rate-Distortion-Accuracy Tradeoff: JPEG Case Study) built a fully differentiable proxy of the JPEG pipeline (including the Discrete Cosine Transform and quantization steps) to optimize image-specific quantization tables, maximizing classification accuracy while minimizing file size.
The Quantization Mismatch Gap
While continuous proxies allow networks to train successfully, they introduce a secondary problem: the quantization mismatch gap. Because the network optimizes over a smooth approximation during training but encounters rigid, discrete values during inference, its performance can degrade at deployment.
To address this discrepancy, Wei Wang developed Online Meta Adaptation for Variable-Rate Learned Image Compression. This approach accepts the training-time proxy as an initial approximation, but introduces a test-time adaptation step. By fine-tuning a small set of meta-parameters on the actual, discretely quantized representations during inference, the framework bridges the proxy gap without requiring any additional data overhead for transmission.
Key Foundational Work
End-to-end optimization of nonlinear transform codes for perceptual quality (2016) — contributed 1 concept elements to this direction.
Postprocessing of Compressed Images via Sequential Denoising (2015) — contributed 1 concept elements to this direction.