In traditional video compression, we avoid sending every pixel of every frame by predicting the current frame from the previous one and only transmitting the "residual"—the difference between the prediction and reality. While this first-order prediction is the bedrock of codecs like H.264, a new wave of research suggests we are leaving massive efficiency gains on the table.
The core insight is that the auxiliary data we use to compress video—motion vectors, residual maps, and even 3D geometry—is itself highly predictable over time. By applying predictive coding to these internal signals, researchers are achieving dramatic bitrate reductions across video, 3D scenes, and even distributed machine learning.
The Intuition of Second-Order Redundancy
Most neural video codecs treat motion vectors and residual maps as independent entities for each frame. However, as Reza Pourreza and colleagues at Qualcomm AI Research point out in Boosting neural video codecs by exploiting hierarchical redundancy, motion is typically smooth. If an object is moving left in frame 10, it is likely moving left in frame 11.
This is "second-order redundancy." Instead of encoding a full motion map for every frame, we can use a lightweight predictor to extrapolate the current motion from the past. We then only transmit the residual of the motion map. This approach, also championed by Oren Rippel in the ELF-VC framework, allows for "motion-compensated motion compensation," drastically lowering the bitrate required for auxiliary data.
Residuals of Residuals
Even the errors in our predictions (the residuals) aren't random. In Learned Video Compression with Residual Prediction and Loop Filter, Chao Liu demonstrates that motion prediction errors exhibit significant temporal correlation. If a codec struggles to predict a complex texture in one frame, it will likely struggle in the next.
By introducing a Residual Prediction Network (RP-Net), the system predicts the current frame's residual from previously reconstructed residuals. This treats the residual stream as a temporally correlated sequence rather than a series of isolated "error images." Similarly, Goluck Konuko and Stéphane Lathuilière show in RDAC that animation-based residuals are highly consistent over time. Encoding only the temporal difference between these residuals can save over 70% in bitrate for specific tasks like talking-head videos.
Predictive Coding in 3D and Volumetric Space
The principle of predictive coding extends naturally to dynamic 3D data, where the "motion" is the displacement of vertices or points. Tingyu Fan introduced D-DPCC, the first end-to-end framework for dynamic point cloud compression, which performs motion estimation directly in a latent feature space to encode only the geometric residuals.
This is particularly effective in the emerging field of 4D Gaussian Splatting. In 4DGC, Qiang Hu and Li Song use a multi-resolution motion grid to predict how 3D primitives transform over time. Instead of storing the full state of every Gaussian for every frame, they encode a reference keyframe and the subsequent motion vectors. By storing only the residuals between predicted and actual positions, systems like HiFi4G can reduce the storage footprint of high-fidelity human performances to less than 2MB per frame.
Zero-Bitrate Reference Generation
One of the most aggressive applications of this idea is "in-loop frame prediction." As explored in the ALVC framework, the decoder can use historical motion patterns to synthesize a high-quality reference frame at zero bit-rate cost.
Because the decoder has access to the same reconstructed history as the encoder, it can run its own internal "hallucination" of the next frame. The encoder then only needs to send the correction (the residual) to fix the decoder's guess. This shifts the burden from the communication channel to the local computation at the decoder, which is ideal for low-bandwidth streaming.
Beyond Video: Federated Learning and Analytics
Predictive coding is not limited to pixels or geometry; it is a general strategy for any time-varying signal. In Communication-Efficient Federated Learning via Predictive Coding, Kai Yue and Huaiyu Dai apply this to distributed machine learning. Since model updates in federated learning are highly correlated over training rounds, they transmit only the "residual error" between the actual update and a shared prediction, slashing communication overhead by up to 99%.
Similarly, for edge video analytics, Jiawei Shao and Jun Zhang developed TOCOM-TEM, which uses a temporal entropy model to predict current feature distributions from past ones. By transmitting only the residual information relative to these semantic predictions, the system maintains high accuracy for tasks like object detection while using a fraction of the bandwidth of traditional video codecs.