Abstract
Video understanding is traditionally computationally expensive due to the high-dimensional nature of video data and the quadratic complexity of standard Transformer architectures. This paper presents Video RWKV, a novel framework utilizing the LSTM-CrossRWKV (LCR) unit. By combining the linear complexity of the RWKV architecture with LSTM-style recurrence and an innovative "edge-prompting" mechanism, Video RWKV achieves competitive performance on action recognition benchmarks with a fraction of the computational cost of attention-based models.
Introduction
Processing video sequences presents a significant challenge: 3D CNNs often struggle with long-range temporal dependencies, while Transformers encounter a quadratic complexity bottleneck ($O(N^2)$) relative to sequence length. Given the high redundancy in video frames, efficient processing is crucial for real-world applications. Video RWKV addresses this by leveraging the Receptive Weighted Key Value (RWKV) architecture, which offers the parallelizable training of Transformers alongside the efficient, linear-complexity inference typical of RNNs.
Methodology: The LSTM-CrossRWKV (LCR) Framework
The core innovation is the LCR unit, designed to capture temporal dynamics efficiently.
Key Components
- Cross RWKV Gate: This module facilitates interaction between current frame features and historical information using a linear-complexity bidirectional attention mechanism to aggregate inter-frame features over time.
- Edge Prompt Learning: To focus the model on relevant motion, the Canny operator (with thresholds determined by Otsu’s method) extracts edge maps. These edges serve as "prompts," guiding the model's attention toward the subject of the action and away from static background noise.
- Enhanced LSTM Recurrence: The LCR unit integrates an LSTM-like execution path where extracted edge information acts as a forgetting gate, helping manage long-term memory by filtering redundant spatial-temporal data.
- Tube Masking: A masking strategy is employed to further reduce redundancy and prevent overfitting by limiting the processing of "filler" information.
Experiments and Results
The framework was evaluated on three major benchmarks: Kinetics-400 (K400), Something-Something V2 (SSv2), and Jester.
Performance on Jester
The LCR model demonstrated exceptional efficiency on the Jester dataset, achieving a 90.83% Top-1 accuracy. Notably, it achieved this with only 5.14M parameters and 0.022 Tflops, representing a massive reduction in computational requirements compared to models like TimeSformer-L (43.1 Tflops).
Key Insights
- Structural Guidance: Using edge data as a gating mechanism allows the model to naturally ignore static backgrounds, focusing instead on the moving contours of the actor.
- Efficiency at Scale: By maintaining $O(N)$ complexity, the architecture avoids the memory explosion typical of Transformers in long-form video analysis.
- Hybrid Architecture: The "spatial mixing" module within the recurrent step effectively blends global attention strengths with recurrent efficiency.
Limitations and Future Work
While efficient, the model's reliance on classical recurrent structures can create parallelization bottlenecks during training compared to pure CNNs. Future research will explore scaling LCR to "foundation-level" video models and its application in generative tasks like video prediction and generation.
Summary updated