In neural image compression, the goal is to transform an image into a compact latent representation and then encode that representation into as few bits as possible. The efficiency of this process is governed by a fundamental principle of information theory: the number of bits required to store a value is inversely proportional to the probability assigned to it by the model.
As David Minnen et al. argue, the bitrate is fundamentally limited by the accuracy of the entropy model—the component responsible for estimating the probability distribution of these latent representations. If the model’s estimated distribution deviates from the true distribution of the data, the resulting bitrate increases, creating a gap between actual performance and the theoretical compression limit.
The Failure of the Single Gaussian Assumption
Early learned compression models often relied on a simple, fixed probability distribution—typically a single Gaussian—to model latent variables. While computationally efficient, this assumption creates a significant "distribution mismatch." Real-world image data is rarely so simple; it is highly irregular, multi-modal, and varies significantly across different regions of an image.
Zhengxue Cheng et al. and Jooyoung Lee et al. point out that a single bell curve is too restrictive to capture the complex statistics of image latents. For instance, a latent variable might represent a feature that could belong to several different textures. A single Gaussian would attempt to "average" these possibilities, resulting in a wide distribution with low peak probability. This inaccuracy forces the entropy coder to use more bits than necessary to resolve the uncertainty.
Multi-modal and Flexible Likelihoods
To bridge this gap, researchers have moved toward more flexible likelihood models that can adapt to the "peaky" or multi-modal nature of latent data.
- Gaussian Mixture Models (GMMs): By using a weighted sum of multiple Gaussian distributions, models can represent several distinct "peaks" of probability. Zhengxue Cheng et al. and Heming Sun et al. demonstrated that discretized GMMs allow for much more precise entropy estimation, particularly for complex or lossless compression tasks.
- Generalized Gaussian Models (GGMs): Recognizing that latents often exhibit heavy tails or extreme sharpness at zero, Haotian Zhang et al. proposed the GGM. By adding a single "shape" parameter, the model can morph between Laplacian and Gaussian shapes, offering a balance between the flexibility of GMMs and the speed of simpler models.
- Hybrid Mixtures: Some approaches, such as the one proposed by Haisheng Fu et al., combine Gaussian, Laplacian, and Logistic distributions to capture the nuances of latent features at high bitrates.
Contextual and Hierarchical Refinement
Even a flexible distribution is only useful if the model knows which parameters (mean, variance, shape) to use for a specific pixel. This is where context comes in. Modern entropy models use two primary sources of information to refine their predictions:
- Hyperpriors (Side Information): The encoder sends a small amount of extra data—a "summary" of the latents—to help the decoder set up the correct probability distributions. David Minnen et al. showed that this hierarchical approach is essential for capturing global structure.
- Autoregressive Context: The model looks at previously decoded neighboring pixels to predict the current one. Jooyoung Lee et al. combined this "bit-free" spatial context with hyperpriors to significantly boost efficiency.
However, standard autoregressive models are notoriously slow because they must process pixels one by one. To solve this, Kede Ma et al. introduced techniques like "code dividing" and 3D zigzag scanning to enable parallel decoding while maintaining context sensitivity.
Capturing Global and Structural Dependencies
Recent work has expanded the scope of context beyond immediate neighbors. Atefeh Khoshkhahtinat et al. utilized Transformers to capture long-range dependencies across the latent space, while Jingbo Lu et al. introduced a learnable dictionary that "memorizes" common structural patterns from the training data. By comparing the current latent to this dictionary, the model can leverage global knowledge to further sharpen its probability estimates.
Ultimately, the evolution of this field is a move away from rigid assumptions toward models that can dynamically adapt their "expectations" to the specific content of an image. As the mismatch between the assumed and actual distributions shrinks, neural codecs move closer to the theoretical limits of data compression.
Key Foundational Work
Joint Autoregressive and Hierarchical Priors for Learned Image Compression (2018) — contributed 1 concept elements to this direction.
Multi-scale and Context-adaptive Entropy Model for Image Compression (2019) — contributed 1 concept elements to this direction.
Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression (2019) — contributed 1 concept elements to this direction.