End to End Differentiable Video Compression

36 concepts28 papers1 proposal

The Modular Bottleneck of Traditional Codecs

Traditional video compression standards—such as H.264, HEVC (H.265), and VVC (H.266)—rely on a hand-engineered paradigm known as hybrid video coding. To compress a video frame, these codecs split the image into a grid of rigid blocks, search for matching blocks in previously decoded frames (motion estimation), transmit the spatial displacement vectors (motion compensation), and then compress the remaining error (residual coding) using discrete mathematical transforms like the Discrete Cosine Transform (DCT).

While highly optimized, this architecture suffers from a fundamental mathematical limitation: its components are optimized locally and heuristically. The motion estimation module operates independently of the transform module, and the block-partitioning scheme relies on combinatorial searches that are computationally exhausting. As Alan Bovik demonstrated in Speeding up VP9 Intra Encoder with Hierarchical Deep Learning Based Partition Prediction, traditional encoders spend massive computational effort searching for optimal block splits because the underlying modules cannot communicate or adapt dynamically. Because these hand-crafted pipelines are non-differentiable, they cannot be optimized globally from end to end.

End-to-end neural video compression solves this by replacing the entire modular pipeline with a single, continuous deep neural network. By making every stage of the codec differentiable, the entire system can be optimized simultaneously using backpropagation, allowing the encoder, decoder, and entropy models to work in perfect harmony.

Global Rate-Distortion Optimization

The core reason neural video compression works so effectively is that it replaces local heuristics with a unified, global loss function: the rate-distortion cost.

$$L = R + \lambda D$$

Here, $R$ represents the estimated bit rate required to transmit the compressed data, and $D$ represents the distortion (reconstruction error) between the original and decoded frames. In a traditional codec, balancing $R$ and $D$ requires complex, hand-tuned rate control algorithms. In a fully differentiable pipeline, however, gradients flow backward from the final decoded pixels all the way to the initial motion layers.

As shown by Guo Lu et al. in DVC: An End-To-End Deep Video Compression Framework, a neural network can ingest raw video pixels and jointly optimize motion estimation, motion compensation, and residual compression as a single architecture. If the network makes a poor motion prediction, the resulting residual becomes complex and expensive to encode, driving up the rate $R$. Because the pipeline is differentiable, the network learns to adjust its motion estimation parameters to find representations that are inherently easier to compress, achieving an optimal balance that hand-tuned modules cannot replicate.

From Rigid Blocks to Continuous Flow

Traditional codecs are notorious for "blocky" artifacts, especially at low bitrates. This happens because rigid block matching cannot accurately model complex, fluid real-world motion, such as a swirling cloud of dust or a camera panning across a textured surface.

Neural codecs eliminate these boundaries by replacing block matching with pixel-wise continuous motion fields, typically modeled via optical flow. However, predicting exact pixel movement is challenging when objects deform or suffer from motion blur. To address this, Eirikur Agustsson et al. introduced a powerful generalization in Scale-Space Flow for End-to-End Optimized Video Compression.

Instead of forcing the network to predict a strict spatial displacement for every pixel, the authors added a "scale" (or blur) parameter to the optical flow field. When the network is uncertain about where a pixel has moved—such as during a rapid camera transition or a blur—it adaptively applies a scale-space blur to that region. This allows the network to gracefully trade off precision for compressibility, preventing the generation of high-frequency residual errors that would otherwise waste valuable bits.

Feature-Domain Perception and Temporal Memory

Early iterations of neural codecs focused primarily on compressing raw pixel residuals. Modern architectures, however, achieve far greater efficiency by shifting the entire compression process from the pixel domain into a latent feature domain.

In FLAVC: Learned Video Compression with Feature Level Attention, Chun Zhang et al. demonstrated that transitioning to global feature-domain perception allows networks to bypass heavy, pixel-level motion estimation entirely. By utilizing a mixed Transformer-CNN encoder, the model captures global context and structural relationships across frames. This feature-level understanding enables the codec to recognize that an object has merely shifted or changed lighting, allowing it to reconstruct high-fidelity frames with massive bitrate savings.

Furthermore, video possesses deep temporal correlations that extend far beyond adjacent frames. To exploit this long-range context, Fabian Mentzer et al. integrated recurrent structures into both the autoencoder and the entropy estimation layers in Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model. By maintaining a continuous recurrent state, the model retains a memory of past visual elements, meaning it does not need to repeatedly transmit redundant texture information over time.

To maximize the utility of this temporal memory, Ren Yang et al. proposed a hierarchical quality framework in Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement. The system intentionally compresses specific "anchor" frames at a higher quality. These high-fidelity anchors then serve as reliable references that propagate through the recurrent network, enhancing the reconstruction quality of all subsequent intermediate frames.

Reaching Codec Parity and Beyond

The evolution of end-to-end neural video compression has followed a rapid trajectory from theoretical viability to state-of-the-art dominance. Early milestones, such as the interpolation-based codec proposed by Chao-Yuan Wu et al. in Video Compression Through Image Interpolation and the PixelMotionCNN framework by Zhibo Chen et al. in Learning for Video Compression, proved that learned models could successfully compete with established standards like H.264.

Recently, the field has crossed a critical performance threshold. Yibo Shi et al. demonstrated in AlphaVC: High-Performance and Efficient Learned Video Compression that a fully end-to-end neural codec can comprehensively outperform VVC (H.266), the most advanced traditional standard, achieving significant bitrate reductions across diverse datasets.

As detailed by Chuanmin Jia et al. in Emerging Advances in Learned Video Compression: Models, Systems and Beyond, this transition from pixel-level residual coding to sophisticated feature-domain conditional models has solidified learned compression as a superior alternative to hand-engineered frameworks. These global optimization capabilities are no longer just academic exercises; they are currently driving international standardization efforts, such as the MPAI-EEV project highlighted by Chuanmin Jia et al. in MPAI-EEV: Standardization Efforts of Artificial Intelligence based End-to-End Video Coding, fundamentally redefining how visual data is transmitted across the globe.

Key Foundational Work

Video Compression Through Image Interpolation (2018) — contributed 1 concept elements to this direction.
Restoration by Compression (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_end_to_end_neural_video_compression_2e4ead2b,
  title = {end-to-end neural video compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-neural-video-compression/rdr_2e4ead2b8e50402bb940b2fc6beda828},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_neural_video_compression_2e4ead2b_v1,
  title = {end-to-end neural video compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-neural-video-compression/rdr_2e4ead2b8e50402bb940b2fc6beda828/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake