Fused behavior recognition model based on attention mechanism

Lei Chen, Rui Liu, Dongsheng Zhou, Xin Yang, Qiang Zhang

Introduction

Human behavior recognition is a critical component in various AI domains, including security surveillance, healthcare monitoring, and autonomous driving. Despite significant progress, two primary challenges persist: the trade-off between model complexity and computational speed, and the difficulty in distinguishing between visually similar actions. For instance, actions such as "waving" vs. "clapping" or "flic-flac" vs. "cartwheel" often lead to high misclassification rates in standard models.

Figure 1: Examples of actions that are easily confused in behavior recognition datasets, such as flic-flac vs. cartwheel and wave vs. clap.

Methodology: The Res34-SE-IM-Net Architecture

To address these issues, this research introduces Res34-SE-IM-Net, a hybrid model that optimizes both spatial and temporal feature extraction.

2D and 3D Feature Fusion

Rather than relying solely on computationally intensive 3D Convolutional Neural Networks (CNNs), the proposed architecture utilizes a "top-heavy" approach. A modified 2D ResNet34 first extracts spatial features from individual video frames. These features are then stacked and processed by a 3D-ResNet18 to capture temporal dynamics. This hybrid strategy maintains high accuracy while significantly reducing the computational load.

Squeeze-and-Excitation (SE) Attention Mechanism

The core innovation for disambiguating similar actions is the integration of Squeeze-and-Excitation (SE) modules. These modules act as attention gates, recalibrating channel-wise feature responses by explicitly modeling interdependencies between channels. By assigning higher weights to distinguishing features and suppressing noise, the model gains the nuance required to separate "confusing" actions. The SE-IM-BasicBlock further incorporates identity mapping to ensure stable gradient flow and feature preservation.

Experimental Evaluation

The model was evaluated on three benchmark datasets:

  • HMDB51: 51 action categories including "jump" and "ride bike."
  • UCF101: 101 categories representing a wide variety of human activities.
  • Something-Something v1: 174 classes focusing on human-object interactions, which requires deep temporal understanding.

To enhance efficiency, the model employs a sparse sampling strategy, processing only 16 frames per video, which is sufficient for high-level recognition while drastically boosting inference speed.

Results and Discussion

The Res34-SE-IM-Net achieved state-of-the-art performance across multiple metrics:

  • Accuracy: The model reached 92.196% Top-1 accuracy on UCF101 and 71.85% on HMDB51. On the challenging Something-Something v1 dataset, it achieved a competitive 41.398% Top-1 accuracy.
  • Computational Efficiency: The model processes between 18.8 and 20.2 Videos Per Second (VPS). This is approximately 40 times faster than the standard I3D model (~0.5 VPS), making it suitable for real-time deployment on standard hardware.
  • Confusion Reduction: A key finding was the impact of the attention mechanism; the confusion between "wave" and "clap" was reduced by 21%, validating the effectiveness of the SE module in fine-grained recognition.

Conclusion

The Res34-SE-IM-Net provides a robust framework for real-time behavior recognition by effectively fusing 2D/3D convolutions with attention mechanisms. While it excels in standard benchmarks, future work will focus on developing temporal attention modules to better capture the complex object-human interactions found in datasets like Something-Something v1 where the order of frames is paramount.

Summary updated

Abstract

With the rapid development of deep learning technology, behavior recognition based on video streams has made great progress in recent years. However, there are also some problems that must be solved: (1) In order to improve behavior recognition performance, the models have tended to become deeper, wider, and more complex. However, some new problems have been introduced also, such as that their real-time performance decreases; (2) Some actions in existing datasets are so similar that they are difficult to distinguish. To solve these problems, the ResNet34-3DRes18 model, which is a lightweight and efficient two-dimensional (2D) and three-dimensional (3D) fused model, is constructed in this study. The model used 2D convolutional neural network (2DCNN) to obtain the feature maps of input images and 3D convolutional neural network (3DCNN) to process the temporal relationships between frames, which made the model not only make use of 3DCNN’s advantages on video temporal modeling but reduced model complexity. Compared with state-of-the-art models, this method has shown excellent performance at a faster speed. Furthermore, to distinguish between similar motions in the datasets, an attention gate mechanism is added, and a Res34-SE-IM-Net attention recognition model is constructed. The Res34-SE-IM-Net achieved 71.85%, 92.196%, and 36.5% top-1 accuracy (The predicting label obtained from model is the largest one in the output probability vector. If the label is the same as the target label of the motion, the classification is correct.) respectively on the test sets of the HMDB51, UCF101, and Something-Something v1 datasets.
BibTeX
@article{chen2020fused,
  author = {Lei Chen and Rui Liu and Dongsheng Zhou and Xin Yang and Qiang Zhang},
  title = {{Fused behavior recognition model based on attention mechanism}},
  journal = {Vis. Comput. Ind. Biomed. Art},
  year = {2020},
  doi = {10.1186/s42492-020-00045-x},
}
Figure