Traditional video compression relies on a simple premise: consecutive frames look mostly the same. To exploit this temporal redundancy, codecs use motion estimation to track how pixels move from one frame to the next. Instead of saving every frame from scratch, the encoder transmits a set of motion vectors and a "pixel residual"—the remaining difference between the motion-compensated prediction and the actual frame.
However, this approach introduces a new problem: the motion vectors themselves require a significant number of bits to transmit. As video resolutions and framerates scale up, the data overhead of these raw motion maps becomes a major bottleneck.
Predictive motion coding solves this by recognizing a deeper layer of structure: motion itself is continuous and highly predictable. If an object moves smoothly across the screen, its velocity and direction do not change randomly from frame to frame. By predicting the current frame's motion vectors from historical trajectories or scene context, codecs can transmit only a "motion residual" (or omit motion data entirely), unlocking massive bitrate savings.
Exploiting Second-Order Redundancy
In standard video coding, motion vectors are treated as independent metadata used to predict pixels. Predictive motion coding introduces the concept of second-order redundancy—the predictable patterns found within the motion maps themselves. Because physical objects obey the laws of inertia, motion fields exhibit high temporal correlation.
[Frame t-2 Motion] \
--> [Trajectory Predictor] --> [Predicted Motion t]
[Frame t-1 Motion] / |
v
[Actual Motion t]
|
v
[Motion Residual Δt]
Instead of encoding raw displacement vectors, modern architectures extrapolate the current frame's motion from past frames and transmit only the correction delta. The effectiveness of this strategy scales directly with video framerate, as higher framerates exhibit smoother and more predictable motion patterns (Pourreza et al.).
This mechanism has been integrated into both traditional and learned video codecs through several distinct approaches:
- Optical Flow Residuals: In learned video compression, Rippel et al. demonstrate in ELF-VC: Efficient Learned Flexible-Rate Video Coding that predicting the current frame's optical flow from previous frames and transmitting only a residual correction and a mask significantly reduces the motion bit overhead.
- Multi-Frame Trajectories: Rather than relying on just the immediate past frame, Wu introduces a multi-frame approach in M-LVC: Multiple Frames Prediction for Learned Video Compression, tracking the trajectory of motion vectors across multiple preceding frames to predict the current motion field.
- Recurrent Contextual Learning: In the standardization efforts of MPAI-EEV: Standardization Efforts of Artificial Intelligence based End-to-End Video Coding, Jia et al. utilize recurrent neural units to learn temporal context from previous frames, allowing the codec to transmit only the residual difference between predicted and actual motion.
By strategically investing bits into refining motion accuracy, encoders can drastically reduce the entropy of the final pixel residual signal, optimizing the overall rate-distortion performance (Lin).
Context-Conditioned Motion Coding
Rather than simply subtracting a predicted motion vector from an actual one, advanced neural codecs frame motion coding as a conditional generation or estimation task. By conditioning the probability distribution of the current motion data on the high-dimensional features of previously decoded frames, the network can compress motion information much more efficiently.
Ho et al. implement this paradigm in CANF-VC: Conditional Augmented Normalizing Flows for Video Compression by using extrapolated flow maps from preceding frames to guide the conditional compression of current motion vectors. This ensures cross-domain consistency between the pixel and motion representations.
Similarly, Li et al. introduce Motion Information Propagation for Neural Video Compression, demonstrating that propagating high-dimensional features from the current frame's context to serve as a predictive prior for the next frame's motion coding reduces the bitrate required specifically for motion data by nearly 24%.
[Decoded Frame Context] ---> [Feature Propagation] ---> [Conditional Prior]
|
v
[Actual Motion Vectors] ---------------------------------> [Entropy Coder] ---> [Compressed Bitstream]
For bi-directional coding (B-frames), Li et al. show in Bi-Directional Deep Contextual Video Compression that predicting current motion vectors from the motion history of reference frames allows the system to encode only the residual motion difference, mitigating the heavy signaling overhead typically associated with bi-directional references.
Decoder-Side Derivation and Zero-Bit Prediction
The ultimate realization of predictive motion coding is the complete elimination of motion vector transmission. If the decoder can independently derive or extrapolate the motion field using only the historical data it has already decoded, the motion data cost drops to zero bits.
In traditional video standards, this concept appeared as localized search seeding. Baek et al. developed a scheme where decoder-side motion estimation is initialized using motion vectors from co-located blocks in preceding frames, scaled by their temporal distance. This restricts the decoder's search to a highly localized range, drastically reducing computational overhead while maintaining prediction accuracy. In hardware implementations, Zhu et al. utilize temporal motion vector derivation for secondary reference frames to predict search centers, bypassing redundant full-window search operations.
In deep learning architectures, this concept enables "zero-bit" reference frame generation. Yang demonstrates in Advancing Learned Video Compression With In-Loop Frame Prediction that recurrent networks can generate high-quality frame predictions from historical temporal data at zero bit-rate cost, shifting the transmission burden entirely to correcting minor residual errors.
Taking this further for bi-directional frames, Liu introduces IBVC: Interpolation-driven B-frame Video Compression. This framework completely eliminates dedicated motion estimation and compensation modules for B-frames, relying instead on the inherent predictive power of deep interpolation networks to handle motion implicitly, focusing bits exclusively on fixing interpolation artifacts.
Broadening the Predictive Paradigm
The core mechanics of predictive motion coding—exploiting temporal continuity to transmit residuals or conditional updates—extend naturally beyond standard 2D video pixels to other dynamic data modalities:
- 3D Point Clouds: Fan et al. apply this to spatial geometry in D-DPCC: Deep Dynamic Point Cloud Compression via 3D Motion Prediction, performing 3D motion estimation and compensation directly in a latent feature space to compress only the dynamic geometric residuals.
- LiDAR Sensor Streams: Song et al. exploit temporal redundancy in sequential LiDAR data by calculating optical flow between consecutive frames and warping historical feature maps to the current time step, minimizing the data needed to describe new sensor observations.
- Stereo Video: Chen et al. combine temporal and cross-view correlations in LSVC: A Learning-based Stereo Video Compression Framework to predict motion and disparity offsets, compressing only the resulting residual to leverage the geometric consistency of calibrated sensors.
Ultimately, these methods rely on the fact that the relationship between motion and the remaining signal errors is highly structured. As demonstrated in Hierarchical Autoregressive Modeling for Neural Video Compression, explicitly conditioning the probability distribution of the residual on the transmitted motion information significantly improves entropy coding efficiency. This structured dependency capitalizes on the inherent correlation between motion complexity and prediction errors, proving that motion and pixels are best coded not as independent streams, but as deeply intertwined facets of a single temporal continuum.
Key Foundational Work
A MFCC-based CELP speech coder for server-based speech recognition in network environments (2005) — contributed 1 concept elements to this direction.
Video coding using a deformation compensation algorithm based on adaptive matching pursuit image decompositions (2003) — contributed 1 concept elements to this direction.