Neural Probability Estimation for Entropy Coding

58 concepts50 papers1 proposal

Data compression is traditionally viewed as a problem of finding clever ways to pack files into fewer bits. However, information theory reveals a deeper truth: compression is fundamentally about prediction.

According to Shannon’s source coding theorem, if we know the exact conditional probability $P(x)$ of a symbol occurring given its context, the mathematically optimal way to encode that symbol requires exactly $-\log_2 P(x)$ bits. If our probability model is perfect, we can compress data down to its true entropy limit.

Traditional entropy coders—such as Arithmetic Coding or Asymmetric Numeral Systems (ANS)—are deterministic algorithms that are excellent at converting these raw probabilities into a compact bitstream. But they are only as good as the probabilities they are given. Historically, codecs relied on hand-crafted, low-order statistical models (like frequency tables or short-context Markov chains) to guess these probabilities.

The core insight of neural entropy coding is a clean separation of concerns: decouple probability estimation from bit assignment. By replacing rigid, hand-crafted statistical tables with high-capacity neural networks, we can use deep learning as a powerful "probability oracle" to predict data distributions with unprecedented accuracy, leaving the final bit-packing to standard entropy coders.

The Core Mechanism: Decoupling Prediction from Coding

To understand why this works, consider a sequence of data points. Instead of hardcoding rules about what patterns usually follow one another, we pass the history of the sequence into a neural network. The network outputs a probability distribution over all possible next values (for instance, via a softmax layer).

[ Past Context ] ---> [ Neural Network ] ---> [ Predicted Probability P(x) ] ---> [ Arithmetic Coder ] ---> [ Compressed Bits ]

Because the encoder and decoder are deterministic and share the exact same neural network weights, the decoder can reconstruct the sequence step-by-step. It uses the already-decoded history to run the exact same neural network forward pass, regenerating the identical probability distribution needed to decode the next bits.

This framework was pioneered early on by Schmidhuber et al., who demonstrated that training a neural network to minimize prediction error allows it to approximate the true conditional expectations of character sequences. When paired with an arithmetic coder, this neural predictor easily outperformed traditional dictionary-based methods like Lempel-Ziv algorithms by capturing much higher-order text dependencies.

Why It Works: The Equivalence of Learning and Compressing

The reason neural networks make such spectacular compressors becomes obvious when we look at how they are trained. When an autoregressive model (like a Transformer) learns to predict the next token in a sequence, it minimizes the cross-entropy loss:

$$\mathcal{L} = -\sum P_{\text{true}}(x) \log_2 P_{\text{pred}}(x)$$

This loss function is literally measuring the theoretical bit-rate required to encode the data under the model's predicted distribution. Minimizing cross-entropy loss is mathematically identical to maximizing compression efficiency.

This equivalence implies that any state-of-the-art generative model is, out of the box, a world-class compressor. As shown by Deletang et al., large language models trained purely on text can be repurposed as general-purpose compressors for entirely different modalities. By feeding raw byte streams from images or audio files into a language model and passing its output logits to an arithmetic coder, the model achieves compression ratios that rival or beat domain-specific codecs like PNG and FLAC.

Because these models possess a deep, internalized prior of structural patterns, they can be applied to highly specialized domains. For example, Kalathil et al. utilized large language models to push the boundaries of lossless text compression, while Wang et al. demonstrated that off-the-shelf language models can act as zero-shot statistical priors to losslessly compress complex neural network gradients during distributed training.

Managing Context: Autoregressive vs. Hyperprior Models

When applying neural entropy coding to high-dimensional data like images, video, or audio, modeling the probability of every single raw pixel or sample autoregressively becomes computationally prohibitive. To solve this, modern systems typically compress data in two distinct ways:

Backward-Adaptive (Autoregressive Context)

This approach predicts the probability of a current symbol based on its causal spatial or temporal neighbors. For instance, Toderici et al. introduced a secondary recurrent neural network specifically to capture spatial dependencies within the binary latent codes of an image compressor, demonstrating that latent representations still contain significant redundant patterns that can be exploited for bit savings.

Forward-Adaptive (Hyperpriors)

To avoid the sequential decoding bottleneck of autoregressive models, forward-adaptive systems use a "hyperprior" network. The encoder extracts a compact set of side-information (the hyperprior) that describes the distribution (e.g., the mean and variance) of the primary latent features. This side-information is compressed and sent along with the bitstream. Wu et al. utilized this strategy in video compression, employing a hyperprior autoencoder to model the Laplacian probability distributions of motion and residual data.

To maximize efficiency, state-of-the-art architectures often combine both concepts. Yang et al. and Fu integrated global hyperpriors with local autoregressive context models, allowing the entropy coder to benefit from both global structural guidance and immediate local correlations.

                  +------------------------+
                  |  Hyperprior Network    | (Global Statistics)
                  +-----------+------------+
                              |
                              v
[ Latent Features ] ---> [ Entropy Model ] <--- [ Autoregressive Context ] (Local Neighbors)
                              |
                              v
                     [ Predicted P(x) ]

Adapting to Non-Sequential and Complex Topologies

One of the greatest strengths of neural probability estimation is its flexibility. Traditional compression algorithms struggle with unstructured or non-sequential data, but neural networks can be tailored to match the underlying geometry of virtually any data source.

3D and Geometric Data

For sparse 3D point clouds, Huang et al. and Que et al. replaced fixed statistical tables with deep neural networks designed to predict the occupancy probabilities of nodes within hierarchical octree structures. Similarly, in 3D Gaussian Splatting, Chen et al. leveraged a structured hash grid to provide spatial context for predicting attribute distributions, while Liu et al. serialized unstructured 3D data into long-context sequences to exploit spatial redundancies via attention mechanisms.

Graph Topologies

Network traffic data does not fit nicely onto a 1D timeline or a 2D pixel grid. To compress network logs losslessly, Almasan et al. implemented Spatio-Temporal Graph Neural Networks (ST-GNNs). The network learns the complex, shifting behavioral distributions of data routing across a physical topology, providing highly accurate probability distributions to an arithmetic coder.

Ultimately, neural entropy coding works because it shifts the burden of compression from rigid algorithmic design to statistical learning. By framing compression as a task of maximizing predictive confidence, any advancement in deep learning architectures directly translates into smaller file sizes and more efficient data transmission.

Key Foundational Work

Predictive Coding with Neural Nets: Application to Text Compression (1994) — contributed 3 concept elements to this direction.
Full Resolution Image Compression with Recurrent Neural Networks (2017) — contributed 2 concept elements to this direction.
LW-FQZip 2: a parallelized reference-based compression of FASTQ files (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_entropy_coding_c7ff36bd,
  title = {neural entropy coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-entropy-coding/rdr_c7ff36bd68a345ed82ed5d6c6a8bdaf2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_entropy_coding_c7ff36bd_v1,
  title = {neural entropy coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-entropy-coding/rdr_c7ff36bd68a345ed82ed5d6c6a8bdaf2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake