Abstract
Efficient long-short temporal modeling is key for enhancing the performance of action recognition tasks. In this paper, we propose a new two-stream action recognition network, termed as MENet, consisting of a Motion Enhancement (ME) module and a Video-level Aggregation (VLA) module to achieve long-short temporal modeling. Specifically, motion representations have been proved effective in capturing short-term and high-frequency action. However, current motion representations are calculated from adjacent frames, which may have poor interpretation and bring useless information (noisy or blank). Thus, for short-term motions, we design an efficient ME module to enhance the short-term motions by mingling the motion saliency among neighboring segments. As for long-term aggregations, VLA is adopted at the top of the appearance branch to integrate the long-term dependencies across all segments. The two components of MENet are complementary in temporal modeling. Extensive experiments are conducted on UCF101 and HMDB51 benchmarks, which verify the effectiveness and efficiency of our proposed MENet.
Introduction
Action recognition in video is a balancing act between two types of information: appearance (what is in the frame) and motion (how things move over time). Traditionally, researchers have relied on "Optical Flow" to capture motion, but calculating it is incredibly slow and computationally expensive. Simpler alternatives like "RGBDiff" (subtracting one frame from the next) are fast but often produce noisy, "blank," or uninterpretable feature maps. Furthermore, many models struggle to connect information across an entire video, focusing only on short snippets and missing the "big picture" of the action.
MENet is a novel two-stream architecture that achieves state-of-the-art action recognition by efficiently modeling both "micro" and "macro" time scales. By replacing computationally expensive optical flow with a high-speed Motion Enhancement (ME) module and capturing long-term context via a zero-parameter Video-level Aggregation (VLA) module, the system runs up to 100x faster than traditional methods while maintaining top-tier accuracy.
The MENet Framework
The authors propose MENet, which splits the task into two specialized branches:
1. Motion Enhancement (ME) Module
Instead of calculating complex pixel-level trajectories, the ME module uses a clever max-pooling strategy on adjacent frames. It "amplifies" motion boundaries and entity contours. This highlights the most important parts of a movement—such as the edge of an arm moving—while ignoring static background noise. The authors realized that for action recognition, a perfect vector of every pixel's movement isn't necessary; capturing how the contours of an object change is sufficient.
2. Video-level Aggregation (VLA) Module
To handle long-term dependencies, the authors use a "temporal channel shift" technique at the top of the appearance branch. This allows the network to "peek" at features from preceding and succeeding segments without adding any new parameters to the model. This zero-cost "memory" enables the model to understand the temporal sequence of the entire video.
3. Two-Stream Fusion and Initialization
The model combines high-frequency motion cues from the ME branch with high-level semantic context from the appearance branch. A key innovation is Cross-Branch Initialization: the appearance branch is initialized using weights learned from the motion branch. This "compels" the appearance branch to focus on moving entities rather than getting distracted by static backgrounds.
Experimental Results
The model was evaluated on industry-standard benchmarks: UCF101 (101 categories, 13,320 clips) and HMDB51 (51 classes, 6,766 videos), with large-scale pre-training on Kinetics400.
- Blazing Speed: The ME module achieves a processing speed of 1500 FPS, which is 100 times faster than traditional TV-L1 optical flow methods.
- High Accuracy: On UCF101, MENet achieved 95.6% accuracy, and on HMDB51, it reached 73.5%. These results are comparable to or better than much more complex, computationally expensive state-of-the-art models like TSM and ECO.
- Ablation Success: Adding the VLA module and the specific transfer-learning strategy improved accuracy by over 3% without increasing the computational cost.
Conclusion
MENet demonstrates that efficient long-short temporal modeling is possible without the heavy overhead of optical flow. While highly effective on short-to-medium length clips, future work may explore how this architecture scales to extremely long-form video and the use of even lighter backbones for edge device deployment.
Summary updated