Variable Rate Neural Compression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2244 papers370 conceptsfrozen 2026-10-04

Traditional neural codecs suffer from a "practicality gap": while they outperform classical standards like JPEG or H.265 in quality, they typically require a separate set of trained weights for every target bitrate. This creates a massive storage and deployment burden, as a service provider would need to store dozens of models to handle varying network conditions.

Variable rate neural compression solves this by enabling a single model to traverse the entire rate-distortion curve. Instead of training discrete models, researchers have developed mechanisms to modulate the information density of a single latent representation on the fly.

Scaling the Latent Space with Gain Units

The most intuitive way to control bitrate is to treat the latent representation like a volume knob. In G-VAE, Ze Cui and colleagues introduced "gain units"—simple scaling layers that multiply the latent channels by a learned factor before quantization.

When the scaling factor is small, the values are pushed toward zero and "crushed" by the quantizer, resulting in fewer bits. When the factor is large, more detail is preserved. This approach allows a single model to achieve continuous bitrate adaptation by simply adjusting these scaling factors at inference time. Jona Ballé further formalized this in Nonlinear Transform Coding, showing that parameterizing network weights as a function of the rate-distortion tradeoff allows a model to navigate the curve smoothly.

Hierarchical Quantization and Bitstream Truncation

For audio and speech, the field has converged on Residual Vector Quantization (RVQ). In the EnCodec architecture by Alexandre Défossez and Yossi Adi, the encoder doesn't produce one code, but a hierarchy of them. Each successive layer of the quantizer encodes the "residual" (the error) left by the previous layer.

This creates a naturally scalable bitstream. If the network is congested, the system can simply stop sending the later, "finer" codes. The decoder still reconstructs the audio using only the "coarse" codes. To make this robust, Rithesh Kumar and the team behind Improved RVQGAN utilized "quantizer dropout" during training, forcing the model to be accurate even when only a subset of the codebooks is available.

Adaptive Feature Modulation for Video

Video compression adds the complexity of temporal "drift"—errors that accumulate over time. Oren Rippel and Alexander G. Anderson addressed this in ELF-VC by conditioning the entire network on a quality-level map. By feeding the desired bitrate directly into the model as an input, the network learns to change its internal behavior to match the target.

Building on this, Jiahao Li and Bin Li developed DCVC-FM, which uses "feature modulation" to scale internal latent representations. This allows a single model to cover a quality range of over 11 dB in PSNR, effectively replacing what used to require a dozen separate models.

Token Granularity and Importance Ranking

A more recent trend involves treating media as a sequence of tokens. In Control-GIC, Anqi Li and Huihui Bai proposed adjusting the "granularity" of image patches. Complex areas of an image receive more tokens, while simple areas receive fewer.

Similarly, One-D-Piece by Keita Miwa uses a "Tail Token Drop" strategy. By training the model to prioritize information in the first few tokens of a sequence, the system can dynamically truncate the token stream at inference time to meet a specific file size target. This mirrors how Peitian Zhang uses "Activation Beacons" to compress long LLM contexts, where the model is exposed to various compression ratios during training to ensure it can handle any level of information density at runtime.

Papers in this version

Showing 60 of 244 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 202 papers
  2. Version 1 · 2026-10-04 · 244 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_variable_rate_neural_compression_f18470da,
  title = {variable-rate neural compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-rate-neural-compression/rdr_f18470da4ef649f68233d5cba79f2a46},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variable_rate_neural_compression_f18470da_v1,
  title = {Variable Rate Neural Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-rate-neural-compression/rdr_f18470da4ef649f68233d5cba79f2a46/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}