Long-Short Temporal Modeling for Efficient Action Recognition

Liyu Wu, Yuexian Zou, Can Zhang

TL;DR

MENet is a novel two-stream architecture that achieves state-of-the-art action recognition by efficiently modeling both "micro" and "macro" time scales. By replacing computationally expensive optical flow with a high-speed Motion Enhancement (ME) module and capturing long-term context via a zero-parameter Video-level Aggregation (VLA) module, the system runs up to 100x faster than traditional methods while maintaining top-tier accuracy.

What Problem Does This Paper Address?

Action recognition in video is a balancing act between two types of information: appearance (what is in the frame) and motion (how things move over time).

Traditionally, researchers use "Optical Flow" to capture motion, but calculating it is incredibly slow and computationally "heavy." Simpler alternatives like "RGBDiff" (subtracting one frame from the next) are fast but often produce noisy, "blank," or uninterpretable feature maps. Furthermore, many models struggle to connect the dots across an entire video, focusing only on short snippets and missing the "big picture" of the action.

The authors of this paper set out to create a model that is both efficient (fast enough for real-world use) and effective at modeling both short-term flickers of movement and long-term temporal sequences.

Methods

The authors propose MENet, which splits the task into two branches:

  1. Motion Enhancement (ME) Module: Instead of calculating complex pixel-level trajectories, the ME module uses a clever max-pooling strategy on adjacent frames. It "amplifies" motion boundaries and entity contours. This highlights the most important parts of a movement (like the edge of an arm moving) while ignoring static background noise.
  2. Video-level Aggregation (VLA) Module: To handle long-term dependencies, the authors use a "temporal channel shift" technique at the top of the appearance branch. This allows the network to "peek" at features from preceding and succeeding segments without adding any new parameters to the model.
  3. Two-Stream Fusion: The model combines the high-frequency motion cues from the ME branch with the high-level semantic context from the appearance branch to make a final prediction.

Datasets

The model was evaluated on the industry-standard benchmarks for action recognition:

  • UCF101: 101 action categories (e.g., playing guitar, diving) across 13,320 video clips.
  • HMDB51: A more challenging dataset with 51 classes and 6,766 videos.
  • Kinetics400: Used for large-scale pre-training to give the model a strong foundational understanding of human movement.

Key Insights and Ideas

  • Contours Matter: The authors realized that for action recognition, you don't need a perfect vector of every pixel's movement. You just need to see how the contours of an object change. The ME module focuses specifically on these salient boundaries.
  • Efficiency via Shifting: Long-term modeling doesn't have to be expensive. By simply shifting feature channels across the time dimension (VLA), the model gains a "memory" of the whole video for free (zero extra parameters).
  • Cross-Branch Initialization: A subtle but powerful trick used here is initializing the appearance branch using weights learned from the motion branch. This "compels" the appearance branch to focus on moving entities rather than getting distracted by static backgrounds.

Key Results

  • Blazing Speed: The ME module achieves a processing speed of 1500 FPS, which is 100 times faster than traditional TV-L1 optical flow methods.
  • High Accuracy: On UCF101, MENet achieved 95.6% accuracy, and on HMDB51, it reached 73.5%. These results are comparable to or better than much more complex, computationally expensive state-of-the-art models like TSM and ECO.
  • Ablation Success: Adding the VLA module and the specific transfer-learning strategy improved accuracy by over 3% without increasing the computational "cost" of the model.

Limitations and Future Work

While MENet is highly efficient, the paper primarily focuses on short-to-medium length clips typical of the UCF and HMDB datasets. How this architecture scales to extremely long-form video (minutes or hours) remains an open question. Additionally, while the ME module is 100x faster than optical flow, the overall system still relies on a two-stream backbone (ResNet50); future work could explore even lighter backbones for deployment on mobile or edge devices.

Summary updated

Abstract

Efficient long-short temporal modeling is key for enhancing the performance of action recognition task. In this paper, we propose a new two-stream action recognition network, termed as MENet, consisting of a Motion Enhancement (ME) module and a Video-level Aggregation (VLA) module to achieve long-short temporal modeling. Specifically, motion representations have been proved effective in capturing short-term and high-frequency action. However, current motion representations are calculated from adjacent frames, which may have poor interpretation and bring useless information (noisy or blank). Thus, for short-term motions, we design an efficient ME module to enhance the short-term motions by mingling the motion saliency among neighboring segments. As for long-term aggregations, VLA is adopted at the top of the appearance branch to integrate the long-term dependencies across all segments. The two components of MENet are complementary in temporal modeling. Extensive experiments are conducted on UCF101 and HMDB51 benchmarks, which verify the effectiveness and efficiency of our proposed MENet.
BibTeX
@inproceedings{wu2021long,
  author = {Liyu Wu and Yuexian Zou and Can Zhang},
  title = {{Long-Short Temporal Modeling for Efficient Action Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=5P4N0ygmw6j},
}

Loading related content...

Figure