Abstract
In classic video action recognition, labels may not contain enough information about the diverse video appearance and dynamics, thus, existing models that are trained under the standard supervised learning paradigm may extract less generalizable features. We evaluate these models under a cross-dataset experiment setting, as the above label bias problem in video analysis is even more prominent across different data sources. We find that using the optical flows as model inputs harms the generalization ability of most video recognition models. Based on these findings, we present a multi-task learning paradigm for video classification. Our key idea is to avoid label bias and improve the generalization ability by taking data as its own supervision or supervising constraints on the data. First, we take the optical flows and the RGB frames by taking them as auxiliary supervisions, and thus naming our model as Reversed Two-Stream Networks (Rev2Net). Further, we collaborate the auxiliary flow prediction task and the frame reconstruction task by introducing a new training objective to Rev2Net, named Decoding Discrepancy Penalty (DDP), which constraints the discrepancy of the multi-task features in a self-supervised manner. Rev2Net is shown to be effective on the classic action recognition task. It specifically shows a strong generalization ability in the cross-dataset experiments.
Introduction
In computer vision, generalization—the ability of a model to perform well on unseen data sources—remains a significant challenge. In video action recognition, models often suffer from label bias, where coarse video-level labels fail to capture the intricate nuances of motion and appearance. Traditionally, researchers have addressed this using "two-stream" networks that ingest both raw RGB frames and pre-computed optical flow. However, empirical evidence suggests that optical flow features are often dataset-specific and can actually hinder generalization when a model is tested on a new domain (e.g., transitioning from YouTube clips to cinematic movies).
Methods: Reversed Two-Stream Networks (Rev2Net)
To overcome the limitations of traditional two-stream inputs, we propose the Reversed Two-Stream Network (Rev2Net). The core philosophy of Rev2Net is to treat motion and temporal structure as self-supervised targets rather than explicit inputs.
Multi-Task Learning Framework
Rev2Net utilizes a single encoder (based on the I3D architecture) to process a sequence of 32 RGB frames. During training, the model is optimized across three concurrent tasks:
- Action Classification: Standard supervised learning using video labels.
- Flow Prediction: A self-supervised task where the model generates optical flow from RGB frames.
- Reversed Frame Reconstruction: A self-supervised task requiring the model to reconstruct input frames in reverse order.
Decoding Discrepancy Penalty (DDP)
A key innovation is the Decoding Discrepancy Penalty (DDP). Since flow prediction and frame reconstruction tasks may provide conflicting gradients to the encoder, DDP acts as a collaborative constraint. It minimizes the discrepancy between the two decoder streams using Frobenius norms for low-level features and KL divergence for high-level latent representations, forcing the encoder to learn a more robust, unified representation of the video data.
Experimental Results
We evaluated Rev2Net on major benchmarks including UCF101, HMDB51, and Kinetics. A critical component of our evaluation was the cross-dataset setting, where models were trained on one dataset and tested on another to measure true generalization.
Key Findings
- State-of-the-Art Performance: On the UCF101 benchmark, Rev2Net (using only RGB frames at test time) achieved 94.6% accuracy, significantly outperforming the base I3D model (84.5%).
- Superior Generalization: In cross-dataset experiments (HMDB → UCF), Rev2Net reached 63.1% accuracy, compared to 56.4% for the standard I3D. Overall, the multi-task approach improved classification accuracy by 11.9% to 14.1% in these challenging transfer scenarios.
- Efficiency: Because the decoders are only used during training, Rev2Net requires only RGB frames at inference time, eliminating the need for computationally expensive optical flow pre-computation.
Conclusion
Rev2Net demonstrates that shifting optical flow from an input to a supervision target forces the model to internalize the relationship between appearance and dynamics. By leveraging multi-task learning and the DDP constraint, we produce features that are significantly more portable across different video domains. Future work will explore additional self-supervised tasks and the application of this framework to Transformer-based backbones.
Summary updated