Introduction
Human behavior recognition is a critical component in various AI domains, including security surveillance, healthcare monitoring, and autonomous driving. Despite significant progress, two primary challenges persist: the trade-off between model complexity and computational speed, and the difficulty in distinguishing between visually similar actions. For instance, actions such as "waving" vs. "clapping" or "flic-flac" vs. "cartwheel" often lead to high misclassification rates in standard models.
Methodology: The Res34-SE-IM-Net Architecture
To address these issues, this research introduces Res34-SE-IM-Net, a hybrid model that optimizes both spatial and temporal feature extraction.
2D and 3D Feature Fusion
Rather than relying solely on computationally intensive 3D Convolutional Neural Networks (CNNs), the proposed architecture utilizes a "top-heavy" approach. A modified 2D ResNet34 first extracts spatial features from individual video frames. These features are then stacked and processed by a 3D-ResNet18 to capture temporal dynamics. This hybrid strategy maintains high accuracy while significantly reducing the computational load.
Squeeze-and-Excitation (SE) Attention Mechanism
The core innovation for disambiguating similar actions is the integration of Squeeze-and-Excitation (SE) modules. These modules act as attention gates, recalibrating channel-wise feature responses by explicitly modeling interdependencies between channels. By assigning higher weights to distinguishing features and suppressing noise, the model gains the nuance required to separate "confusing" actions. The SE-IM-BasicBlock further incorporates identity mapping to ensure stable gradient flow and feature preservation.
Experimental Evaluation
The model was evaluated on three benchmark datasets:
- HMDB51: 51 action categories including "jump" and "ride bike."
- UCF101: 101 categories representing a wide variety of human activities.
- Something-Something v1: 174 classes focusing on human-object interactions, which requires deep temporal understanding.
To enhance efficiency, the model employs a sparse sampling strategy, processing only 16 frames per video, which is sufficient for high-level recognition while drastically boosting inference speed.
Results and Discussion
The Res34-SE-IM-Net achieved state-of-the-art performance across multiple metrics:
- Accuracy: The model reached 92.196% Top-1 accuracy on UCF101 and 71.85% on HMDB51. On the challenging Something-Something v1 dataset, it achieved a competitive 41.398% Top-1 accuracy.
- Computational Efficiency: The model processes between 18.8 and 20.2 Videos Per Second (VPS). This is approximately 40 times faster than the standard I3D model (~0.5 VPS), making it suitable for real-time deployment on standard hardware.
- Confusion Reduction: A key finding was the impact of the attention mechanism; the confusion between "wave" and "clap" was reduced by 21%, validating the effectiveness of the SE module in fine-grained recognition.
Conclusion
The Res34-SE-IM-Net provides a robust framework for real-time behavior recognition by effectively fusing 2D/3D convolutions with attention mechanisms. While it excels in standard benchmarks, future work will focus on developing temporal attention modules to better capture the complex object-human interactions found in datasets like Something-Something v1 where the order of frames is paramount.
Summary updated