Reversing Two-Stream Networks with Decoding Discrepancy Penalty for Robust Action Recognition

Yunbo Wang, Zhiyu Yao, Hongyu Zhu, Mingsheng Long, Jianmin Wang, Philip S. Yu

Abstract

In classic video action recognition, labels may not contain enough information about the diverse video appearance and dynamics, thus, existing models that are trained under the standard supervised learning paradigm may extract less generalizable features. We evaluate these models under a cross-dataset experiment setting, as the label bias problem in video analysis is even more prominent across different data sources. We find that using optical flows as model inputs harms the generalization ability of most video recognition models. Based on these findings, we present a multi-task learning paradigm for video classification. Our key idea is to avoid label bias and improve generalization by taking data as its own supervision. We utilize optical flows and RGB frames as auxiliary supervisions, naming our model Reversed Two-Stream Networks (Rev2Net). Further, we collaborate the auxiliary flow prediction task and the frame reconstruction task by introducing a new training objective, Decoding Discrepancy Penalty (DDP), which constrains the discrepancy of multi-task features in a self-supervised manner. Rev2Net specifically shows a strong generalization ability in cross-dataset experiments.

Introduction: The Challenge of Label Bias

Standard video action recognition models often struggle when deployed in new environments. This failure is frequently attributed to label bias, where training labels (e.g., "archery") fail to capture the full complexity of video dynamics, backgrounds, and camera angles. A common solution is the "two-stream" approach, which feeds both RGB frames and pre-computed optical flow into the model. However, research indicates an "optical flow trap": while motion maps improve performance on specific datasets, they often fail to generalize across different data sources, such as moving from YouTube-based UCF101 to movie-based HMDB51.

The Rev2Net Framework

To address these limitations, we propose the Reversed Two-Stream Network (Rev2Net). Unlike traditional architectures that use optical flow as an input, Rev2Net treats it as a supervisory signal. By shifting the complexity from the input to the supervision side, the model learns more robust spatiotemporal features.

Conceptual comparison between traditional Two-Stream CNNs and the proposed Rev2Net architecture.

Multi-Task Self-Supervision

Rev2Net utilizes a single RGB encoder with three distinct heads:

  1. Action Classification: The standard supervised task.
  2. Optical Flow Prediction: An auxiliary decoder that reconstructs motion maps.
  3. Reversed Frame Reconstruction: A decoder that rebuilds video frames in reverse order, acting as an information bottleneck that forces the model to learn high-level dynamics.

Decoding Discrepancy Penalty (DDP)

To ensure these tasks collaborate effectively, we introduce the Decoding Discrepancy Penalty (DDP). DDP penalizes the distance between the features of the flow and reconstruction decoders. It uses the Frobenius Norm for low-level visual consistency and KL Divergence for high-level conceptual alignment. This prevents "task interference" and ensures a consistent feature space.

Experimental Results

The model was evaluated on UCF101, HMDB51, and Kinetics, with a particular focus on cross-dataset generalization.

Cross-dataset performance and ablation study of Rev2Net on UCF101 and HMDB51.

Superior Generalization

In cross-dataset benchmarks, Rev2Net significantly outperformed standard models like TSN, TSM, and I3D. For instance, when trained on UCF101 and tested on HMDB51, Rev2Net improved the base I3D model's accuracy by 11.9%. In the reverse direction, it saw a 14.1% improvement.

Efficiency and Performance

On standard benchmarks, Rev2Net achieved state-of-the-art results, reaching 97.1% on UCF101 and 78.0% on HMDB51 when using both RGB and Flow for training. Crucially, because the decoders are discarded at test time, Rev2Net maintains the inference speed of a single-stream RGB network while delivering the accuracy of a two-stream model.

Conclusion

Rev2Net demonstrates that optical flow is more effective as a teacher than a crutch. By leveraging self-supervised auxiliary tasks and the DDP loss, we can train models that are both more accurate and more robust to the domain shifts common in real-world video analysis. Future work may extend this multi-task paradigm to other domains like video segmentation and object detection.

Summary updated

Abstract

In classic video action recognition, labels may not contain enough information about the diverse video appearance and dynamics, thus, existing models that are trained under the standard supervised learning paradigm may extract less generalizable features. We evaluate these models under a cross-dataset experiment setting, as the above label bias problem in video analysis is even more prominent across different data sources. We find that using the optical flows as model inputs harms the generalization ability of most video recognition models. Based on these findings, we present a multi-task learning paradigm for video classification. Our key idea is to avoid label bias and improve the generalization ability by taking data as its own supervision or supervising constraints on the data. First, we take the optical flows and the RGB frames by taking them as auxiliary supervisions, and thus naming our model as Reversed Two-Stream Networks (Rev2Net). Further, we collaborate the auxiliary flow prediction task and the frame reconstruction task by introducing a new training objective to Rev2Net, named Decoding Discrepancy Penalty (DDP), which constraints the discrepancy of the multi-task features in a self-supervised manner. Rev2Net is shown to be effective on the classic action recognition task. It specifically shows a strong generalization ability in the cross-dataset experiments.
BibTeX
@inproceedings{wang2020reversing,
  author = {Yunbo Wang and Zhiyu Yao and Hongyu Zhu and Mingsheng Long and Jianmin Wang and Philip S. Yu},
  title = {{Reversing Two-Stream Networks with Decoding Discrepancy Penalty for Robust Action Recognition}},
  booktitle = {ICME},
  year = {2020},
  doi = {10.1109/icme46284.2020.9102724},
}
Figure