Flexible Rate Architectures for Neural Data Compression

349 concepts241 papers

In traditional data compression codecs like JPEG or HEVC, adjusting the bitrate is trivial: you simply turn a dial to change the quantization parameter. However, early deep learning-based compressors lacked this basic flexibility.

To understand why, consider how a standard neural autoencoder is trained. The network minimizes a joint rate-distortion loss function:

$$L = R + \lambda D$$

Here, $R$ represents the estimated bit budget (entropy), $D$ represents the reconstruction error, and $\lambda$ is a fixed Lagrange multiplier that dictates the trade-off. Because $\lambda$ is hardcoded into the loss function during training, the network weights become permanently optimized for that single, specific point on the rate-distortion curve.

As reviewed by Siwei Ma et al. in Image and Video Compression With Neural Networks: A Review, this constraint creates a severe deployment bottleneck. To support a wide range of network bandwidths or storage limits, service providers historically had to train, store, and deploy entirely separate models for each target bitrate. This multi-model overhead is highly impractical for memory-constrained environments like IoT devices (Yu Mao et al. in Easz: An Agile Transformer-based Image Compression Framework for Resource-constrained IoTs) or hardware-integrated video decoders (Xiaodan Song et al. in A practical convolutional neural network as loop filter for intra frame).

To bridge this gap, researchers have developed flexible-rate architectures that break the one-model-per-rate paradigm. These methods generally fall into three intuitive categories: dynamic conditioning, progressive refinement, and slimmable capacity.


Dynamic Conditioning: Passing the Rate as an Input

If a fixed $\lambda$ binds a network to a single bitrate, the most direct solution is to treat $\lambda$ as a dynamic input variable rather than a training constant.

In Variable Rate Deep Image Compression With a Conditional Autoencoder, the authors introduce a framework where both the Lagrange multiplier $\lambda$ and the quantization bin size $\Delta$ are fed directly into the network alongside the image. By exposing these parameters to the model during training across a distribution of values, the network learns a continuous space of transforms.

Image (x) ───► [ Conditional Encoder ] ───► Quantization (Δ) ───► Latents (z)
                      ▲
                      │
               Target Rate (λ, Δ)

To make the internal features responsive to these inputs, architectures rely on feature modulation layers. For instance, Mu-Jung Chen et al. utilize an Adaptive Feature Modulation module in Learned Hierarchical B-frame Coding with Adaptive Feature Modulation for YUV 4: 2: 0 Content to scale and shift intermediate feature maps dynamically based on the desired rate.

This conditioning can also be applied unevenly across an image. Noor Fathima Ghouse et al. extended this concept to spatial rate-distortion control in A neural video codec with spatial rate-distortion control. By conditioning the network on both a global rate parameter and a spatial importance map, a single model can allocate more bits to regions of interest (such as faces in a video call) while compressing backgrounds aggressively.


Progressive Refinement: Truncating the Bitstream

Another approach avoids choosing a specific rate prior to encoding. Instead, it structures the compressed data so that it can be cut off at any arbitrary point during transmission—a property known as Fine Granular Scalability.

Recurrent Neural Networks (RNNs) naturally support this behavior by encoding data iteratively. As demonstrated by George Toderici et al. in Variable Rate Image Compression with Recurrent Neural Networks George Toderici and Full Resolution Image Compression with Recurrent Neural Networks, an LSTM architecture can compress an image by repeatedly calculating and encoding the residual error left over from the previous iteration.

Iteration 1: [Input] ────────► [RNN Encoder] ──► Bits (Low Quality)
                                    │
                               Residual Error
                                    ▼
Iteration 2: [Residual] ──────► [RNN Encoder] ──► Bits (Medium Quality)
                                    │
                               Residual Error
                                    ▼
Iteration 3: [Residual] ──────► [RNN Encoder] ──► Bits (High Quality)

The decoder processes these bit packets sequentially. If the network connection drops or storage is constrained, the decoder simply stops executing further iterations, degrading the quality gracefully. This recurrent approach was later optimized by Nick Johnston et al. in Improved Lossy Image Compression With Priming and Spatially Adaptive Bit Rates for Recurrent Networks by introducing hidden-state priming to carry over contextual information between iterations.

Rather than using recurrent loops, progressive scalability can also be achieved by organizing the latent space itself. In DPICT: Deep Progressive Image Compression Using Trit-Planes, Jae-Han Lee et al. decompose continuous latent features into ordered ternary digit planes ("trit-planes"). Because these planes are sorted by their rate-distortion importance, the resulting bitstream can be truncated at any byte boundary to match fluctuating network conditions perfectly.


Quantization Independence and Slimmable Capacity

A third angle of attack focuses on decoupling the underlying mathematical transform from the quantization step entirely, or scaling the network's physical size alongside the bitrate.

Thierry Dumas explores this decoupling in Autoencoder Based Image Compression: Can the Learning be Quantization Independent?. The author demonstrates that an autoencoder can be trained to learn a representation where the quantization step size is modified exclusively at test time. This proves that a neural network can learn a generic, high-fidelity transform space that does not inherently depend on a fixed rate-distortion trade-off.

However, changing the bitrate often changes the required computational complexity; decoding a low-bitrate thumbnail shouldn't require the same hardware power as a high-fidelity 4K frame. To address this, Zhaocheng Liu et al. introduced the Slimmable Video Codec. By utilizing slimmable neural network layers, a single model can dynamically adjust its internal channel widths. This allows the system to scale down both its bitrate and its computational footprint simultaneously on resource-constrained hardware.

Similarly, in the domain of generative models, Zhengang Li 0001 developed SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model. By training a single "supernet" that contains multiple embedded sub-networks, a single deployment can adapt on the fly to varying resolution and hardware constraints, eliminating multi-model storage bloat entirely.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 250 papers
  2. Version 1 · 2026-10-04 · 240 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_rate_model_overhead_faff7365,
  title = {fixed-rate model overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-rate-model-overhead/rdr_faff7365374d4a3a94df6438deb36c68},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_fixed_rate_model_overhead_faff7365_v2,
  title = {fixed-rate model overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-rate-model-overhead/rdr_faff7365374d4a3a94df6438deb36c68/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}

See something wrong? Report a mistake