Random Temporal Skipping for Multirate Video Analysis

Yi Zhu, Shawn Newsam

Abstract

Current state-of-the-art approaches to video understanding adopt temporal jittering to simulate analyzing the video at varying frame rates. However, this does not work well for multirate videos, in which actions or subactions occur at different speeds. The frame sampling rate should vary in accordance with the different motion speeds. In this work, we propose a simple yet effective strategy, termed random temporal skipping (RTS), to address this situation. This strategy effectively handles multirate videos by randomizing the sampling rate during training. It is an exhaustive approach, which can potentially cover all motion speed variations. Furthermore, due to the large temporal skipping, our network can see video clips that originally cover over 100 frames. Such a time range is enough to analyze most actions/events. We also introduce an occlusion-aware optical flow learning method that generates improved motion maps for human action recognition. Our framework is end-to-end trainable, runs in real-time, and achieves state-of-the-art performance on six widely adopted video benchmarks.

Introduction

Most deep learning models for video analysis assume a fixed frame rate, typically sampling a short snippet of consecutive frames to identify an action. However, real-world video is rarely consistent. This paper addresses two main challenges: Action Speed Variance, where different individuals perform the same action at different speeds, and Transmission Issues, such as packet drops in surveillance footage that lead to jerky, missing frames. Standard temporal jittering fails to capture these variations, causing performance to drop when models trained on dense data encounter sparse or variable-rate video.

Methodology

The proposed framework utilizes a three-pronged approach to build a robust action recognition system:

  1. Random Temporal Skipping (RTS): Instead of a fixed stride, RTS randomizes the skip distance between consecutive frames during training. This acts as temporal data augmentation, forcing the network to learn features invariant to motion speed.
  2. Occlusion-Aware MotionNet: To avoid the computational bottleneck of traditional optical flow, the authors use a CNN (MotionNet) to estimate motion from RGB frames. An occlusion-aware loss function, based on forward-backward consistency, helps the model ignore areas where objects disappear, resulting in cleaner motion maps.
  3. Compact Bilinear Encoding: The video is divided into segments, and features are aggregated using Compact Bilinear Pooling. This creates a high-dimensional descriptor that captures the entire video's context more effectively than simple pooling methods.
The proposed framework architecture, integrating Spatial and Temporal CNNs with MotionNet and Bilinear Encoding.

Experimental Results

The framework was evaluated on six benchmarks, including trimmed (UCF101, HMDB51, Kinetics) and untrimmed videos (ActivityNet, VIRAT). The system achieved state-of-the-art performance, notably reaching 96.4% accuracy on UCF101 and 91.1% on ActivityNet.

A key finding was the robustness provided by RTS. When tested on videos with simulated frame-rate changes, models trained without RTS saw significant performance degradation, whereas RTS-enabled models remained stable. Furthermore, the system is highly efficient; by replacing pre-computed optical flow with MotionNet, it runs in real-time and is approximately 14 times faster than heavy-duty models like I3D.

Performance comparison showing the consistent accuracy gains provided by RTS across different sampling strategies.

Conclusion and Future Work

RTS serves as a powerful regularizer, allowing models to see a wider temporal context—sometimes exceeding 100 frames. While performance gains level off when the maximum skip distance exceeds 6 frames, the method remains a significant step forward for real-time, robust video analysis. Future research may explore integrating Recurrent Neural Networks (RNNs) for even longer-range dependencies and applying the framework to online event detection in continuous streams.

Summary updated

Abstract

Current state-of-the-art approaches to video understanding adopt temporal jittering to simulate analyzing the video at varying frame rates. However, this does not work well for multirate videos, in which actions or subactions occur at different speeds. The frame sampling rate should vary in accordance with the different motion speeds. In this work, we propose a simple yet effective strategy, termed random temporal skipping, to address this situation. This strategy effectively handles multirate videos by randomizing the sampling rate during training. It is an exhaustive approach, which can potentially cover all motion speed variations. Furthermore, due to the large temporal skipping, our network can see video clips that originally cover over 100 frames. Such a time range is enough to analyze most actions/events. We also introduce an occlusion-aware optical flow learning method that generates improved motion maps for human action recognition. Our framework is end-to-end trainable, runs in real-time, and achieves state-of-the-art performance on six widely adopted video benchmarks.
BibTeX
@misc{zhu2018random,
  author = {Yi Zhu and Shawn Newsam},
  title = {{Random Temporal Skipping for Multirate Video Analysis}},
  year = {2018},
  eprint = {1810.12522},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.12522},
}
Figure