Abstract
This research explores the critical role of temporal information in enhancing video understanding, specifically focusing on action recognition and semantic segmentation. For action recognition, the "Hidden Two-Stream Network" is introduced to learn motion representations end-to-end, eliminating the need for costly optical flow pre-computation. For semantic segmentation, a framework leveraging video prediction models synthesizes training data, significantly improving model accuracy and robustness. These contributions demonstrate that mining temporal cues is essential for advanced visual perception.
Introduction
Video understanding extends beyond static image analysis by incorporating the temporal dimension. Capturing how objects move and interact over time is vital for tasks like action recognition and autonomous driving. However, traditional methods often struggle with computational efficiency and data scarcity. This paper addresses these challenges by rethinking motion representation and data augmentation through temporal modeling.
Hidden Two-Stream Networks for Action Recognition
Traditional action recognition models utilize a two-stream architecture: one for spatial appearance (RGB) and another for temporal motion (optical flow). While effective, pre-computing optical flow is computationally expensive and requires significant storage.
The proposed Hidden Two-Stream Network integrates a sub-network called MotionNet. Instead of relying on external flow solvers, MotionNet learns to estimate motion in an unsupervised manner by reconstructing frames. This approach allows the model to be trained end-to-end, making the motion representation task-specific. Consequently, the system focuses on motion relevant to the action (e.g., a golf swing) rather than irrelevant background noise. This method is approximately 10x faster than traditional two-stage methods, achieving over 100 frames per second (FPS) on a single GPU.
Enhancing Robustness and Generalization
To address the variability in action speeds, the research introduces Random Temporal Skipping (RTS). By randomly skipping frames during training, the model becomes invariant to different frame rates, acting as a powerful regularizer. Furthermore, for zero-shot action recognition—where the model must identify unseen categories—Universal Representation Learning (URL) is employed. Using Non-negative Matrix Factorization (NMF) with Jensen-Shannon Divergence, the model identifies shared semantic building blocks across different action classes, allowing it to generalize to new datasets without re-training.
Video Prediction for Semantic Segmentation
Semantic segmentation requires pixel-level annotations, which are notoriously labor-intensive to produce. To overcome this "data hunger," the dissertation proposes using video prediction models to propagate labels from annotated frames to adjacent unannotated frames.
To mitigate errors caused by prediction distortions at object boundaries, a Boundary Label Relaxation loss is introduced. This "soft" labeling approach makes the training process robust to the noise inherent in synthesized data. By scaling the training set through this automated propagation, the models achieve superior performance without requiring architectural changes. This approach reached a record mIoU of 83.5% on the Cityscapes dataset.
Key Insights and Results
- Motion is Task-Specific: Lower error in optical flow (EPE) does not always lead to better action recognition. Task-oriented motion representations focus on the actor rather than the background.
- Joint Propagation: When synthesizing data, propagating both the image and the label together ensures alignment, which is more critical for network training than the absolute visual quality of the predicted frame.
- Performance: The framework achieved top-tier results on UCF101 (96.4%) and HMDB51 (72.5%) while maintaining real-time inference speeds.
Conclusion and Future Work
This work highlights that temporal information is a rich source of supervision. By integrating motion learning and leveraging video continuity for data synthesis, we can build video understanding systems that are both more efficient and more accurate. Future research will look toward integrating monocular depth, human pose, and long-term temporal dependencies (such as RNNs or LSTMs) to capture actions that span longer durations.
Summary updated