Mitigating Temporal Drift in Neural Video Compression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2124 papers176 conceptsfrozen 2026-10-04

In video compression, the most efficient way to save space is to avoid repeating information. Instead of storing every frame as a full image, codecs store a "reference" frame and then only record the changes (the residuals) for subsequent frames. This is known as predictive coding.

However, this efficiency comes with a hidden cost: temporal drift. Because each new frame is built upon the reconstructed version of the previous one, any tiny error introduced during compression—like a slight blur or a quantization artifact—doesn't just stay in that frame. It becomes part of the foundation for the next frame, which adds its own errors, creating a "snowball effect" of degradation.

The Recursive Error Problem

The fundamental equation of predictive video coding is $Reconstruction_t = Prediction(Reconstruction_{t-1}) + Residual_t$.

As Guo Lu and colleagues noted in their seminal work on DVC, this recursive dependency means that reconstruction inaccuracies serve as the basis for all future predictions. In a long sequence, these small errors accumulate and amplify, leading to a gradual decline in visual fidelity. This is why a video might start crisp but slowly become "mushy" or filled with "ghosting" artifacts over time.

In traditional codecs like H.264 or H.265, this is managed by periodically inserting "Intra-frames" (I-frames)—complete images that act as a hard reset. But I-frames are expensive in terms of data. The goal of modern neural video coding (NVC) is to minimize the need for these resets by making the predictive process itself more robust.

Sequence-Aware Training

Early neural codecs were often trained on pairs of frames. The model would look at Frame A, predict Frame B, and the loss function would penalize the difference. The problem, as identified in Content Adaptive and Error Propagation Aware Deep Video Compression, is that this "myopic" training doesn't account for the long-term consequences of a single frame's quality.

To fix this, researchers shifted to sequence-based training. By training on five, ten, or more consecutive frames, the loss function can "see" the drift happening. This forces the encoder to prioritize features that remain stable over time. Zongyu Guo and Runsen Feng demonstrated that modulating the training loss to prioritize later frames in a sequence effectively dampens this error propagation, stabilizing long-term prediction.

From Pixels to Feature Context

Traditional codecs work in the pixel domain, warping blocks of pixels from one frame to the next. Neural codecs have a superpower: they can work in latent feature space.

Instead of just predicting the next set of pixels, models like DCVC-FM by Jiahao Li and Bin Li use "feature modulation." They propagate high-dimensional feature contexts across frames. This allows the network to maintain a "memory" of the scene that is more resilient than raw pixels.

However, even features can get "noisy." To combat this, Chuanbo Tang and Dong Liu proposed Deep Context Modulation (DCMVC). Their approach uses "context compensation" to refine these temporal features, effectively "cleaning" the internal memory of the codec as it moves through the video. This ensures that irrelevant information and inaccuracies from previous frames aren't compounded.

Gathering Long-Term Context

The most recent breakthrough in fighting drift is moving beyond the immediate previous frame. If Frame 100 looks a lot like Frame 1 (perhaps a camera panned away and then back), why should the codec rely on the degraded Frame 99?

Linfeng Qi and the team at Microsoft introduced DCVC-LCG, which uses an intra-cluster searching mechanism. This allows the model to gather relevant features from distant frames in the past. By reaching back to high-quality "anchor" frames, the system can bypass the accumulated noise of the intermediate frames, effectively "healing" the sequence without needing a full I-frame reset.

Why It Works: The Closed-Loop Intuition

The success of these methods relies on a closed-loop feedback architecture. As seen in the Residual Deep Animation Codec (RDAC), the encoder must simulate exactly what the decoder will see.

If the encoder makes a prediction based on the original Frame A, but the decoder only has the compressed (and slightly distorted) Frame A, their versions of the world will diverge. By ensuring the prediction is always based on the "actual reconstructed state" available at the decoder, neural codecs can eliminate the mismatch that triggers the most aggressive forms of temporal drift.

Papers in this version

Showing 60 of 124 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 107 papers
  2. Version 1 · 2026-10-04 · 124 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_error_propagation_prediction_dr_53f71878,
  title = {temporal error propagation (prediction drift)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-error-propagation-prediction-drift/rdr_53f71878657d4d928338ca5eebe4dd9f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_temporal_drift_in_neural_vide_53f71878_v1,
  title = {Mitigating Temporal Drift in Neural Video Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-temporal-drift-in-neural-video-compression/rdr_53f71878657d4d928338ca5eebe4dd9f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}