Abstract
In classic video action recognition, labels may not contain enough information about the diverse video appearance and dynamics, thus, existing models that are trained under the standard supervised learning paradigm may extract less generalizable features. We evaluate these models under a cross-dataset experiment setting, as the label bias problem in video analysis is even more prominent across different data sources. We find that using optical flows as model inputs harms the generalization ability of most video recognition models. Based on these findings, we present a multi-task learning paradigm for video classification. Our key idea is to avoid label bias and improve generalization by taking data as its own supervision. We utilize optical flows and RGB frames as auxiliary supervisions, naming our model Reversed Two-Stream Networks (Rev2Net). Further, we collaborate the auxiliary flow prediction task and the frame reconstruction task by introducing a new training objective, Decoding Discrepancy Penalty (DDP), which constrains the discrepancy of multi-task features in a self-supervised manner. Rev2Net specifically shows a strong generalization ability in cross-dataset experiments.
Introduction: The Challenge of Label Bias
Standard video action recognition models often struggle when deployed in new environments. This failure is frequently attributed to label bias, where training labels (e.g., "archery") fail to capture the full complexity of video dynamics, backgrounds, and camera angles. A common solution is the "two-stream" approach, which feeds both RGB frames and pre-computed optical flow into the model. However, research indicates an "optical flow trap": while motion maps improve performance on specific datasets, they often fail to generalize across different data sources, such as moving from YouTube-based UCF101 to movie-based HMDB51.
The Rev2Net Framework
To address these limitations, we propose the Reversed Two-Stream Network (Rev2Net). Unlike traditional architectures that use optical flow as an input, Rev2Net treats it as a supervisory signal. By shifting the complexity from the input to the supervision side, the model learns more robust spatiotemporal features.
Multi-Task Self-Supervision
Rev2Net utilizes a single RGB encoder with three distinct heads:
- Action Classification: The standard supervised task.
- Optical Flow Prediction: An auxiliary decoder that reconstructs motion maps.
- Reversed Frame Reconstruction: A decoder that rebuilds video frames in reverse order, acting as an information bottleneck that forces the model to learn high-level dynamics.
Decoding Discrepancy Penalty (DDP)
To ensure these tasks collaborate effectively, we introduce the Decoding Discrepancy Penalty (DDP). DDP penalizes the distance between the features of the flow and reconstruction decoders. It uses the Frobenius Norm for low-level visual consistency and KL Divergence for high-level conceptual alignment. This prevents "task interference" and ensures a consistent feature space.
Experimental Results
The model was evaluated on UCF101, HMDB51, and Kinetics, with a particular focus on cross-dataset generalization.
Superior Generalization
In cross-dataset benchmarks, Rev2Net significantly outperformed standard models like TSN, TSM, and I3D. For instance, when trained on UCF101 and tested on HMDB51, Rev2Net improved the base I3D model's accuracy by 11.9%. In the reverse direction, it saw a 14.1% improvement.
Efficiency and Performance
On standard benchmarks, Rev2Net achieved state-of-the-art results, reaching 97.1% on UCF101 and 78.0% on HMDB51 when using both RGB and Flow for training. Crucially, because the decoders are discarded at test time, Rev2Net maintains the inference speed of a single-stream RGB network while delivering the accuracy of a two-stream model.
Conclusion
Rev2Net demonstrates that optical flow is more effective as a teacher than a crutch. By leveraging self-supervised auxiliary tasks and the DDP loss, we can train models that are both more accurate and more robust to the domain shifts common in real-world video analysis. Future work may extend this multi-task paradigm to other domains like video segmentation and object detection.
Summary updated