Video RWKV:Video Action Recognition Based RWKV

Zhuowen Yin, Chengru Li, Xingbo DongarXiv 2024 · DOI · arXiv

Abstract

Video understanding is traditionally computationally expensive due to the high-dimensional nature of video data and the quadratic complexity of standard Transformer architectures. This paper presents Video RWKV, a novel framework utilizing the LSTM-CrossRWKV (LCR) unit. By combining the linear complexity of the RWKV architecture with LSTM-style recurrence and an innovative "edge-prompting" mechanism, Video RWKV achieves competitive performance on action recognition benchmarks with a fraction of the computational cost of attention-based models.

Introduction

Processing video sequences presents a significant challenge: 3D CNNs often struggle with long-range temporal dependencies, while Transformers encounter a quadratic complexity bottleneck ($O(N^2)$) relative to sequence length. Given the high redundancy in video frames, efficient processing is crucial for real-world applications. Video RWKV addresses this by leveraging the Receptive Weighted Key Value (RWKV) architecture, which offers the parallelizable training of Transformers alongside the efficient, linear-complexity inference typical of RNNs.

Methodology: The LSTM-CrossRWKV (LCR) Framework

The core innovation is the LCR unit, designed to capture temporal dynamics efficiently.

The LCR Unit and CrossRWKV architecture. (a) shows the internal structure of the LCR unit, including the Canny edge extractor and the Cross RWKV gate. (b) details the CrossRWKV mechanism.

Key Components

  • Cross RWKV Gate: This module facilitates interaction between current frame features and historical information using a linear-complexity bidirectional attention mechanism to aggregate inter-frame features over time.
  • Edge Prompt Learning: To focus the model on relevant motion, the Canny operator (with thresholds determined by Otsu’s method) extracts edge maps. These edges serve as "prompts," guiding the model's attention toward the subject of the action and away from static background noise.
  • Enhanced LSTM Recurrence: The LCR unit integrates an LSTM-like execution path where extracted edge information acts as a forgetting gate, helping manage long-term memory by filtering redundant spatial-temporal data.
  • Tube Masking: A masking strategy is employed to further reduce redundancy and prevent overfitting by limiting the processing of "filler" information.

Experiments and Results

The framework was evaluated on three major benchmarks: Kinetics-400 (K400), Something-Something V2 (SSv2), and Jester.

Performance on Jester

The LCR model demonstrated exceptional efficiency on the Jester dataset, achieving a 90.83% Top-1 accuracy. Notably, it achieved this with only 5.14M parameters and 0.022 Tflops, representing a massive reduction in computational requirements compared to models like TimeSformer-L (43.1 Tflops).

Model configurations and complexity metrics for various LCR variants.

Key Insights

  • Structural Guidance: Using edge data as a gating mechanism allows the model to naturally ignore static backgrounds, focusing instead on the moving contours of the actor.
  • Efficiency at Scale: By maintaining $O(N)$ complexity, the architecture avoids the memory explosion typical of Transformers in long-form video analysis.
  • Hybrid Architecture: The "spatial mixing" module within the recurrent step effectively blends global attention strengths with recurrent efficiency.

Limitations and Future Work

While efficient, the model's reliance on classical recurrent structures can create parallelization bottlenecks during training compared to pure CNNs. Future research will explore scaling LCR to "foundation-level" video models and its application in generative tasks like video prediction and generation.

Summary updated

Abstract

To address the challenges of high computational costs and long-distance dependencies in exist ing video understanding methods, such as CNNs and Transformers, this work introduces RWKV to the video domain in a novel way. We propose a LSTM CrossRWKV (LCR) framework, designed for spatiotemporal representation learning to tackle the video understanding task. Specifically, the proposed linear complexity LCR incorporates a novel Cross RWKV gate to facilitate interaction be tween current frame edge information and past features, enhancing the focus on the subject through edge features and globally aggregating inter-frame features over time. LCR stores long-term mem ory for video processing through an enhanced LSTM recurrent execution mechanism. By leveraging the Cross RWKV gate and recurrent execution, LCR effectively captures both spatial and temporal features. Additionally, the edge information serves as a forgetting gate for LSTM, guiding long-term memory management.Tube masking strategy reduces redundant information in food and reduces overfitting.These advantages enable LSTM CrossRWKV to set a new benchmark in video under standing, offering a scalable and efficient solution for comprehensive video analysis. All code and models are publicly available.
BibTeX
@misc{yin2024video,
  author = {Zhuowen Yin and Chengru Li and Xingbo Dong},
  title = {{Video RWKV:Video Action Recognition Based RWKV}},
  year = {2024},
  eprint = {2411.05636},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.05636},
}
Figure