When training neural networks on video or sequential sensor data, the default approach is to sample consecutive frames: frame $t$, then frame $t+1$, then frame $t+2$. While intuitive, this consecutive sampling strategy introduces a hidden vulnerability: it tightly couples the model's learned features to the specific frame rate of the camera and the average speed of objects in the training set.
If a model is trained exclusively on 30 frames-per-second (FPS) video where cars move at 30 mph, it learns to associate a specific pixel displacement with normal motion. If the sensor drops frames (reducing the effective frame rate) or encounters a vehicle traveling at 80 mph, the pixel displacement jumps drastically. Because the network has never seen such large spatial displacements over a single-frame step, its motion estimation, tracking, or segmentation capabilities break down.
Decoupling Time via Variable Strides
To break this rigid coupling, a powerful data augmentation strategy has emerged: stochastic temporal sampling (also known as multi-stride temporal augmentation). Instead of training on consecutive pairs $(t, t+1)$, the training pipeline randomly selects frames with a variable temporal gap or stride, feeding the model pairs like $(t, t+k)$, where $k$ is a randomly sampled integer offset.
This simple shift fundamentally changes what the network learns. By varying $k$, a single physical sequence can simulate an infinite variety of motion dynamics. A low value of $k$ simulates slow-moving objects or high-frame-rate sensors, while a high value of $k$ simulates rapid motion, motion blur, or low-frame-rate conditions. As a result, the model learns features that are invariant to specific frame rates or object speeds.
Mitigating Velocity Bias
In real-world datasets—especially those used in autonomous driving—the vast majority of frames contain slow-moving or static objects. Kewei Wang demonstrates in Semi-supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix that this distribution creates a severe bias against fast-moving entities. By sub-sampling input sequences during training, the apparent velocity of objects is artificially increased, forcing the model to learn robust representations for high-speed scenarios.
Similarly, Raphael van Kempen et al. demonstrate in Enhancing Lidar-based Object Detection in Adverse Weather using Offset Sequences in Time that randomly skipping frames forces lidar perception models to learn generalized motion patterns rather than specific frame-to-frame transitions. This stochastic temporal sampling prevents overfitting to fixed sensor sampling rates and improves robustness in adverse weather conditions where sensor data can be noisy or intermittent.
Shifting from Displacement Magnitude to Motion Topology
When a model is exposed to a wide variety of frame-hopping intervals, it can no longer rely on memorizing fixed pixel distances. Instead, it is forced to focus on the structural and topological nature of motion.
- Optical Flow and Tracking: Jisoo Jeong et al. show in Imposing Consistency for Optical Flow Estimation that frame-hopping exposes models to a broader range of displacement magnitudes, significantly enhancing their ability to handle large-scale motion. In visual tracking, Minji Kim et al. apply this in Towards Sequence-Level Training for Visual Tracking, showing that altering the temporal distance between frames simulates rapid motion and blur, making trackers highly resilient to real-world temporal discontinuities. For human pose tracking, Jihye Hwang et al. utilize a multi-stride augmentation strategy in Pose estimator and tracker using temporal flow maps for limbs to capture both subtle, slow movements and rapid, large-scale limb displacements.
- Motion Segmentation: Marc Ellenfeld et al. show in Deep Fusion of Appearance and Frame Differencing for Motion Segmentation that varying the temporal gap between frames during training forces models to learn generalized motion patterns that are invariant to specific object speeds. For event-based sensors, Anton Mitrokhin et al. demonstrate in Learning Visual Motion Segmentation Using Event Surfaces that randomly scaling the temporal axis forces the network to prioritize the underlying geometric topology of motion over specific velocities.
Generalization Across Displacement Scales
In geometric and generative video tasks, multi-stride sampling acts as a scale-invariant regularizer. In video super-resolution, Younghyun Jo et al. show in Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation that varying the temporal sampling interval allows a single model to generalize across both slow and fast-moving objects without relying on explicit motion compensation.
For self-supervised odometry, Bin Li et al. apply this concept in Self-supervised Visual-LiDAR Odometry with Flip Consistency to expose the network to a diverse distribution of motion magnitudes, preventing it from overfitting to specific vehicle speeds.
Frequency Equivariance in Periodic Signals
The intuition behind variable temporal strides extends beyond spatial tracking into the domain of physiological signal extraction, such as remote photoplethysmography (rPPG)—detecting heart rates from subtle skin color changes in video.
Deep learning models in this space frequently suffer from a subtle failure mode: they memorize the average heart rate of the training population, creating a static prior that fails on new datasets. To combat this, Nathan Vance et al. in Promoting Generalization in Cross-Dataset Remote Photoplethysmography and Jeremy Speth et al. in Non-Contrastive Unsupervised Learning of Physiological Signals from Video introduce temporal resampling as a core data augmentation technique.
By artificially speeding up or slowing down the video frames during training, the authors alter the underlying frequency of the periodic pulse signal. This forces the neural network to become equivariant to temporal changes. Instead of memorizing a specific frequency band, the model is forced to learn the actual physiological mechanism of blood flow across a wide spectrum of frequencies, drastically reducing cross-dataset errors.
Key Foundational Work
Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation (2018) — contributed 2 concept elements to this direction.
Learning a real-time generic tracker using convolutional neural networks (2017) — contributed 2 concept elements to this direction.