To compress video efficiently, codecs must eliminate temporal redundancy. Instead of saving every frame from scratch, video compression frameworks typically encode an initial keyframe (an Intra-frame or I-frame) and then predict subsequent frames (Predicted frames or P-frames) by estimating motion and encoding only the remaining "residual" difference.
However, this reliance on temporal context introduces a fundamental vulnerability. As noted by Guo Lu et al. in their work on DVC: An End-To-End Deep Video Compression Framework, predictive coding schemes create a sequential chain of dependencies. If a reference frame contains even a minor reconstruction artifact, that artifact is carried forward into the prediction of the next frame. This phenomenon is known as temporal error propagation or prediction drift.
The Anatomy of Prediction Drift
To understand why errors compound, consider how a standard inter-frame codec operates across a timeline. Let $X_t$ be the true video frame at time $t$. Instead of the true frame, the decoder only has access to a reconstructed reference frame $\hat{X}_{t-1}$, which contains some quantization noise or approximation error $\epsilon_{t-1}$:
$$\hat{X}_{t-1} = X_{t-1} + \epsilon_{t-1}$$
When predicting the next frame, the network applies a motion compensation function $f$ based on estimated motion vectors $M_t$:
$$\tilde{X}_t = f(\hat{X}_{t-1}, M_t) = f(X_{t-1} + \epsilon_{t-1}, M_t)$$
The encoder then compresses the residual difference between the true frame and this prediction ($R_t = X_t - \tilde{X}_t$), introducing a new layer of quantization error $\eta_t$. The final reconstructed frame at time $t$ becomes:
$$\hat{X}_t = f(X_{t-1} + \epsilon_{t-1}, M_t) + \eta_t$$
Because the motion compensation function propagates the previous frame's error, $\epsilon_t$ becomes a compounding function of all historical errors ($\epsilon_{t-1}, \epsilon_{t-2}, \dots$) plus the current frame's quantization noise $\eta_t$.
Over a long Group of Pictures (GOP), these tiny inaccuracies accumulate sequentially. As Yung-Han Ho et al. demonstrate in Learned Video Compression for YUV 4: 2: 0 Content Using Flow-based Conditional Inter-frame Coding, this compounding effect causes severe performance degradation as the distance from the initial keyframe increases. Visually, this drift manifests as persistent blurriness, ghosting artifacts, and a gradual loss of high-frequency details, as observed by Jiahao Li et al. in Neural Video Compression with Feature Modulation.
The Myopia of Frame-by-Frame Optimization
Mitigating prediction drift is challenging because traditional rate-distortion optimization (RDO) evaluates compression efficiency myopically—one frame at a time. However, video frames are organized in a directed acyclic graph (DAG) of dependencies. As Tongda Xu et al. point out in Bit Allocation using Optimization, a naive optimization that treats frames independently ignores a critical ripple effect: allocating fewer bits to an early reference frame shifts the rate-distortion probability distribution for all subsequent frames.
Conversely, investing extra bits into an early anchor frame can drastically reduce the overall distortion of an entire video sequence. As argued by Jian Xiong et al. in Periodic Enhanced Frame Based Long-Short-Term Reference in HEVC for Conference and Surveillance Video Coding, high-quality anchor frames act as stable references that prevent quantization errors from leaking down the stream.
To capture these cascading inter-frame dependencies, Junqi Shi et al. developed a reinforcement learning framework in Reinforced Rate Control for Neural Video Compression via Inter-Frame Rate-Distortion Awareness. By treating rate control as a sequential decision-making problem, the authors demonstrate that encoders can optimize bit allocation across the entire dependency graph rather than maximizing isolated frame-level metrics.
Architectural Solutions to Bound Drift
To prevent prediction drift without relying on frequent, bandwidth-heavy I-frame refreshes, modern neural video codecs (NVCs) employ architectures designed to break or restructure the linear dependency chain.
Long-Term Context Gathering
Instead of limiting motion compensation to the immediate previous frame, models can look further back into the video's history. Linfeng Qi et al. introduce an intra-cluster searching mechanism in Long-Term Temporal Context Gathering for Neural Video Compression to pull high-fidelity features from distant reference frames, bypassing the error accumulation of intermediate frames. Similarly, Fabian Mentzer et al. leverage recurrent auto-encoders in Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model to maintain a persistent internal memory state that preserves long-range temporal correlations.
Explicit Hierarchical Structures
Rather than allowing errors to propagate indefinitely down a linear chain, codecs can organize frames into hierarchical layers. Junqi Liao et al. demonstrate in EHVC: Efficient Hierarchical Reference and Quality Structure for Neural Video Coding that explicit hierarchical reference schemes bound error propagation. High-quality upper-layer frames serve as clean anchors, allowing the system to shed accumulated drift and recover visual fidelity quickly.
Group-Aware Optimization
When adapting neural video codecs to specific video content on the fly, updating model parameters frame-by-frame can introduce systematic drift. To solve this, Zhenghao Chen et al. propose a group-aware optimization strategy in Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression. By optimizing lightweight adapters across entire Groups of Pictures simultaneously, the authors ensure that local frame updates remain temporally consistent, preventing the accumulation of localized reconstruction errors over time.
Key Foundational Work
Robust H.263+ video for real-time Internet applications (2002) — contributed 3 concept elements to this direction.