When training deep neural networks to forecast sequential states—whether the 3D coordinates of a human skeleton, the pixels of a video, or the temperature of a room—models often struggle with a frustrating paradox: they are easily outperformed by a baseline that simply predicts no change at all. In human motion forecasting, Julieta Martinez et al. demonstrated that sophisticated recurrent architectures frequently failed to beat a simple "zero-velocity" baseline. The network wastes its capacity trying to reconstruct the absolute spatial structure of the state, rather than learning how that state evolves over time.
The core intuition of residual state prediction is to change the network's target from the absolute future state $X_{t+1}$ to the incremental change $\Delta X = X_{t+1} - X_t$. By adding a hardcoded residual connection from the current state to the output, the model's default behavior when outputting zeros becomes an identity mapping ($X_{t+1} = X_t$). This simple shift fundamentally alters the optimization landscape. As early as 1998, Radek Grzeszczuk et al. recognized that predicting these incremental changes reduces the dynamic range of the network's outputs, making it significantly easier for neural emulators to learn complex physical transitions. Instead of mapping a high-dimensional state space to another high-dimensional state space, the network learns a local mapping centered around zero.
Preventing Compounding Drift in Autoregressive Rollouts
Why does this matter for long-term forecasting? In an autoregressive setup, where a model's predictions are fed back into itself as inputs for the next time step, small errors inevitably compound. If a model predicts absolute coordinates, a slight miscalculation can cause the geometry to warp, leading to characters that stretch unnaturally or physical simulations that completely diverge.
By framing the task as delta prediction, the model is forced to operate on velocities or displacements. For instance, Ehsan Adeli showed that predicting joint velocities rather than absolute positions drastically reduces cumulative error and eliminates the drifting effect in long-term human pose forecasting. Similarly, Lingwei Dang et al. utilized temporal residual coordinate prediction to simplify the optimization landscape, focusing the network entirely on motion dynamics.
This principle is critical not just for human poses, but for complex physical systems. In Lagrangian solid mechanics simulations, the authors of Diffusion-Based Hierarchical Graph Neural Networks for Simulating Nonlinear Solid Mechanics found that predicting incremental node displacements rather than absolute positions was essential for maintaining long-term simulation stability and preventing physical drift over extended temporal horizons.
Anchoring Structure in Generative Models
The benefits of residual prediction extend directly into generative modeling, particularly where temporal consistency must coexist with a fixed reference structure. Consider the challenge of audio-driven talking head generation. If a model predicts absolute facial landmark coordinates frame-by-frame, the resulting video often suffers from micro-jitters and structural distortion because the network fails to perfectly preserve the speaker's underlying anatomy across frames.
To solve this, frameworks like MoDiTalker, proposed by Seyeon Kim et al., predict the difference between current facial landmarks and a reference frame rather than absolute coordinates. This residual formulation anchors the generation process, ensuring that facial movements remain strictly consistent with the speaker's identity over time.
Similarly, in 3D head animation, Wenxuan Zhang et al. and Xiumei Xie et al. demonstrated that predicting motion residuals relative to an initial frame minimizes jitter and prevents unnatural drift during long, highly expressive sequences like singing. By separating the static reference state from the dynamic motion-induced changes, the model's capacity is entirely dedicated to synthesizing smooth transitions.
Exploiting Temporal Persistence and Baseline States
Beyond physical motion and video generation, the residual framework serves as a powerful inductive bias for systems characterized by temporal persistence or known baseline states. In environmental control systems, Felix Bünning et al. showed that predicting the incremental change in a system's state (such as temperature fluctuations) allows machine learning models to better capture underlying thermal dynamics while reducing cumulative control errors.
When dealing with highly non-stationary time series where distributions shift rapidly, Zhiding Liu et al. introduced a residual statistical modeling approach that predicts the difference between future segment statistics and the global input mean, stabilizing the learning process against shifting scales.
Even in biological domains like single-cell genomics, where the effects of a drug or genetic knockout must be modeled, the authors of PopPert: Population-level Joint-Distribution Modeling for Single-Cell Perturbation Prediction utilized a residual prediction strategy to estimate the delta between control and perturbed distributions. Rather than forcing a transformer to reconstruct an entire cellular state from scratch, the model focuses exclusively on the specific deviations caused by the perturbation, ensuring robust and stable training.
Key Foundational Work
Fast Neural Network Emulation of Dynamical Systems for Computer Animation (1998) — contributed 2 concept elements to this direction.
On Human Motion Prediction Using Recurrent Neural Networks (2017) — contributed 2 concept elements to this direction.