Traditional image and video codecs often treat every pixel or block with a "one-size-fits-all" approach. Whether it is a clear blue sky or a complex forest texture, the quantization parameters are typically applied uniformly across large regions. This is fundamentally inefficient: a smooth gradient requires far fewer bits to represent accurately than a high-frequency edge.
Modern research has shifted toward content-adaptive bit allocation, a family of techniques that dynamically redistribute a bit budget across an image based on local information density. By prioritizing "interesting" regions—those with high entropy, complex textures, or semantic importance—these systems achieve higher perceptual quality at significantly lower bitrates.
The Importance Map as a Bit Budgeter
The core intuition behind adaptive compression is that not all pixels are created equal. To operationalize this, researchers like Mu Li and Wangmeng Zuo proposed using a sub-network to generate a spatial importance map [Learning Convolutional Networks for Content-weighted Image Compression].
This map acts as a local mask that decides the bit depth for different regions. In practice, this often involves "pruning" feature channels in low-saliency areas. During training, the sum of this importance map serves as a differentiable proxy for the total bitrate, allowing the model to learn a rate-distortion trade-off where it "buys" more detail for textures and "saves" on smooth backgrounds [Learning Content-Weighted Deep Image Compression]. Wen-Hsiao Peng and colleagues demonstrated that this spatially varying allocation allows neural codecs to approach the performance of high-efficiency standards like BPG by focusing on perceptually critical details [An Autoencoder-based Learned Image Compressor].
Dynamic Granularity and Quantization
While importance maps decide where to spend bits, dynamic quantization decides how to spend them. Instead of a fixed rounding function, the system adjusts the precision of the latent representation on the fly.
Zongyu Guo and Zhizheng Zhang introduced the "Soft-then-Hard" (STH) strategy to bridge the gap between training (which uses noise to simulate quantization) and testing (which uses actual rounding). By combining this with Scaled Uniform Noise (SUN), they enabled the model to predict spatially varying scaling factors for quantization noise [Soft then Hard: Rethinking the Quantization in Neural Image Compression].
More recently, Anqi Li and Feng Li developed Control-GIC, which uses local information entropy to assign image patches to hierarchical "granularity levels" [Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaption]. This allows a single model to adapt to any bitrate by simply scaling the precision of its patches, ensuring that high-complexity regions always receive the finest representation the budget allows.
Beyond Human Perception: Task-Aware Allocation
Most compression is designed for human eyes, but an increasing amount of video is consumed by machines. Standard codecs often discard high-frequency features that are "invisible" to humans but vital for AI tasks like object detection.
To solve this, Christoph Reich developed a control network that optimizes H.264 parameters specifically for machine vision [Deep Video Codec Control for Vision Models]. Similarly, Uri Gadot proposed a reinforcement learning agent, RL-RC-DoT, which adjusts quantization at the macroblock level to preserve regions critical for downstream AI models while aggressively compressing the rest [RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression].
Applications in Restoration and Super-Resolution
The benefits of content-adaptive quantization extend beyond simple storage. In video restoration, Cong Huang and Dong Liu used adaptive quantization to "purify" features. By applying coarse quantization to smooth areas, they effectively suppressed noise, while fine quantization preserved the high-frequency details needed for reconstruction [Neural Compression-Based Feature Learning for Video Restoration].
In the realm of super-resolution, Mingshen Wang and Zhao Zhang introduced Granular-DQ. They observed that most image regions can be processed at very low precision (e.g., 4-bit) without losing quality, provided that high-precision bits are reserved for critical textures [Thinking in Granularity: Dynamic Quantization for Image Super-Resolution]. This "layer-invariant" approach allows the model to maintain the same bit-width for a patch across all network layers, drastically reducing computational overhead while maintaining sharp edges.