Exploring Temporal Information for Improved Video Understanding

Yi Zhu

Abstract

This research explores the critical role of temporal information in enhancing video understanding, specifically focusing on action recognition and semantic segmentation. For action recognition, the "Hidden Two-Stream Network" is introduced to learn motion representations end-to-end, eliminating the need for costly optical flow pre-computation. For semantic segmentation, a framework leveraging video prediction models synthesizes training data, significantly improving model accuracy and robustness. These contributions demonstrate that mining temporal cues is essential for advanced visual perception.

Introduction

Video understanding extends beyond static image analysis by incorporating the temporal dimension. Capturing how objects move and interact over time is vital for tasks like action recognition and autonomous driving. However, traditional methods often struggle with computational efficiency and data scarcity. This paper addresses these challenges by rethinking motion representation and data augmentation through temporal modeling.

Hidden Two-Stream Networks for Action Recognition

Traditional action recognition models utilize a two-stream architecture: one for spatial appearance (RGB) and another for temporal motion (optical flow). While effective, pre-computing optical flow is computationally expensive and requires significant storage.

The proposed Hidden Two-Stream Network integrates a sub-network called MotionNet. Instead of relying on external flow solvers, MotionNet learns to estimate motion in an unsupervised manner by reconstructing frames. This approach allows the model to be trained end-to-end, making the motion representation task-specific. Consequently, the system focuses on motion relevant to the action (e.g., a golf swing) rather than irrelevant background noise. This method is approximately 10x faster than traditional two-stage methods, achieving over 100 frames per second (FPS) on a single GPU.

Enhancing Robustness and Generalization

To address the variability in action speeds, the research introduces Random Temporal Skipping (RTS). By randomly skipping frames during training, the model becomes invariant to different frame rates, acting as a powerful regularizer. Furthermore, for zero-shot action recognition—where the model must identify unseen categories—Universal Representation Learning (URL) is employed. Using Non-negative Matrix Factorization (NMF) with Jensen-Shannon Divergence, the model identifies shared semantic building blocks across different action classes, allowing it to generalize to new datasets without re-training.

Video Prediction for Semantic Segmentation

Semantic segmentation requires pixel-level annotations, which are notoriously labor-intensive to produce. To overcome this "data hunger," the dissertation proposes using video prediction models to propagate labels from annotated frames to adjacent unannotated frames.

To mitigate errors caused by prediction distortions at object boundaries, a Boundary Label Relaxation loss is introduced. This "soft" labeling approach makes the training process robust to the noise inherent in synthesized data. By scaling the training set through this automated propagation, the models achieve superior performance without requiring architectural changes. This approach reached a record mIoU of 83.5% on the Cityscapes dataset.

Key Insights and Results

  • Motion is Task-Specific: Lower error in optical flow (EPE) does not always lead to better action recognition. Task-oriented motion representations focus on the actor rather than the background.
  • Joint Propagation: When synthesizing data, propagating both the image and the label together ensures alignment, which is more critical for network training than the absolute visual quality of the predicted frame.
  • Performance: The framework achieved top-tier results on UCF101 (96.4%) and HMDB51 (72.5%) while maintaining real-time inference speeds.

Conclusion and Future Work

This work highlights that temporal information is a rich source of supervision. By integrating motion learning and leveraging video continuity for data synthesis, we can build video understanding systems that are both more efficient and more accurate. Future research will look toward integrating monocular depth, human pose, and long-term temporal dependencies (such as RNNs or LSTMs) to capture actions that span longer durations.

Summary updated

Abstract

In this dissertation, I present my work towards exploring temporal information for better video understanding. Specifically, I have worked on two problems: action recognition and semantic segmentation. For action recognition, I have proposed a framework, termed hidden two-stream networks, to learn an optimal motion representation that does not require the computation of optical flow. My framework alleviates several challenges faced in video classification, such as learning motion representations, real-time inference, multi-framerate handling, generalizability to unseen actions, etc. For semantic segmentation, I have introduced a general framework that uses video prediction models to synthesize new training samples. By scaling up the training dataset, my trained models are more accurate and robust than previous models even without modifications to the network architectures or objective functions. I believe videos have much more potential to be mined, and temporal information is one of the most important cues for machines to perceive the visual world better.
BibTeX
@misc{zhu2019exploring,
  author = {Yi Zhu},
  title = {{Exploring Temporal Information for Improved Video Understanding}},
  year = {2019},
  eprint = {1905.10654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.10654},
}
Figure