When a human moves, their physical body obeys the laws of classical mechanics: positions change continuously, and velocities scale smoothly. However, when computer vision models attempt to reconstruct this motion from video, they often treat the video not as a continuous stream, but as a sequence of disconnected, static images. This independent frame-by-frame processing introduces a pervasive artifact: high-frequency temporal jitter.
The Root Cause of Jitter
In monocular 3D pose estimation, a model must resolve severe depth ambiguities and occlusions from a 2D image plane. For any given frame, multiple 3D configurations can explain the 2D projection. If a model optimizes for each frame independently, tiny variations in pixel noise, lighting, or sensor limitations can cause the model's prediction to flip between different mathematically plausible 3D states.
As noted by Helge Rhodin, this causes joint positions to fluctuate rapidly between frames. The resulting motion exhibits massive, unrealistic accelerations that imply physically impossible forces, as described by Davis Rempe et al.. This lack of stability degrades the performance of downstream applications, such as reinforcement learning agents that require stable spatial information to interact with their environment (Emma Cramer et al.).
The Limits of Naive Smoothing and the Footskate Dilemma
A natural first instinct to fix jitter is to apply a standard temporal smoothing filter (such as a Gaussian or Kalman filter) over the predicted joint trajectories. While this successfully dampens high-frequency shaking, it introduces a secondary, equally jarring artifact known as "footskate."
As Jianming Zhang explains, standard temporal smoothing filters prioritize continuous motion across all frames equally. However, physical human motion is non-uniform. When a person walks, their foot must remain completely stationary while supporting body weight. A naive smoothing filter averages these stationary frames with the preceding and succeeding moving frames, causing the foot to unrealistically slide or "skate" across the floor. Resolving temporal inconsistency therefore requires moving beyond blind mathematical smoothing toward models that understand either the sequential context of motion or the underlying physics.
Integrating Sequential Context
To prevent sudden jumps, models must carry information from past frames into the present prediction. By framing pose estimation as a sequential tracking problem rather than an isolated regression task, models can use historical motion to constrain current search spaces.
Early approaches utilized structural tracking and regression forests to maintain continuity across frames (Danhang Tang; Chi Xu et al.). Modern deep learning architectures scale this intuition using recurrent neural networks and Transformers. For instance, Zihao Zhang et al. employ an LSTM-based refinement network to enforce temporal constraints on point cloud sequences, filtering out frame-by-frame noise.
Similarly, Yifeng Jiang et al. demonstrate that effectively integrating historical pose information via Transformers prevents the sudden jumps and erratic joint displacements that occur when a system treats frames in isolation. To handle complex, non-rigid body dynamics more granularly, Srikrishna Karanam et al. divide the body into local parts, assigning dedicated recurrent models to individual limbs to capture distinct motion patterns without global interference.
Physics-Based Dynamics and Visual Anchors
Another powerful strategy forces the model's outputs to conform directly to physical laws or explicit inter-frame visual cues. Instead of relying solely on learned temporal priors, these methods use optimization layers to ensure that the implied forces and velocities are realistic.
The framework by Davis Rempe et al. combines deep contact estimators with centroidal trajectory optimization, explicitly penalizing floor penetration and foot sliding by enforcing zero-velocity constraints during contact phases. Similarly, Soshi Shimada et al. achieve real-time physical plausibility by coupling deep learning pose proposals with a physics-based forward dynamics simulation.
Alternatively, temporal consistency can be anchored using continuous visual signals across frames. Thiemo Alldieck et al. minimize the discrepancy between the observed pixel movement (optical flow) in a video and the flow generated by a rendering of the 3D body model, effectively tying the 3D joint trajectories to the continuous mathematical gradients of the video itself. Along the same lines, Haidong Zhu et al. use the temporal consistency of silhouette sequences to correct noisy, jittery skeleton predictions, leveraging the smooth physical transitions of the body's overall shape.
The Temporal Challenge in Generative Models
The rise of generative frameworks, such as diffusion models, has introduced a new flavor of the temporal consistency problem. When diffusion models are applied to synthesize or complete human motion frame-by-frame, they tend to produce severe jitter.
As explained by Lingting Zhu et al., standard diffusion models sample denoising steps independently for each frame. Because the random noise injected at each time step lacks correlation across the temporal axis, the resulting decoded poses fluctuate wildly from frame to frame. To resolve this, the authors introduce specialized gesture stabilizers and temporal conditioning mechanisms to ensure that the denoising process is correlated across time, yielding smooth and rhythmically synchronized movements.
By shifting from independent frame generation to spatio-temporal conditioning schemes, generative models like the one proposed by Angela Castillo et al. can synthesize long-term dependencies that are both anatomically plausible and temporally stable.