An Action Recognition network for specific target based on rMC and RPN

Mingjie Li, Youqian Feng, Zhonghai Yin, Cheng Zhou, Fanghao Dong, Lin Yuan, Yuhao DongJournal of Physics Conference Series 2019 · DOI

Abstract

Action recognition in complex scenes remains challenging due to background interference and the presence of multiple moving objects. This paper introduces rMC-RPN, a hybrid architecture that combines target detection with action recognition to isolate and analyze specific individuals. By employing a "mixed" convolutional ResNet (rMC-50) that utilizes 2D convolutions for spatial features and 3D convolutions for temporal dynamics, the model achieves an optimal balance between efficiency and accuracy. Integrated with a Region Proposal Network (RPN) for precise target localization, the system reaches processing speeds of up to 200 FPS. Experimental results on the UCF-101 benchmark and custom datasets demonstrate that rMC-RPN significantly outperforms traditional 3D-CNN models in both speed and recognition accuracy.

Introduction

Most traditional action recognition models, such as C3D or Two-Stream networks, analyze entire video frames to classify activities. While effective in controlled environments with a single subject, these models often fail in "cluttered" real-world scenarios where multiple people or distracting moving objects are present. If a system cannot distinguish between the primary "operator" and background interference, its recognition accuracy typically plummets.

To address this, the authors propose treating action recognition as a composite task: first, identifying the specific target through detection, and subsequently analyzing their movements. The primary challenge lies in achieving this integration efficiently enough for real-time applications without the massive computational overhead typically associated with full 3D convolutional neural networks (3D-CNNs).

Proposed Methods

The core innovation of the rMC-RPN architecture is its strategic balance of spatial and temporal processing through a multi-stage pipeline.

Mixed Convolutional ResNet (rMC-50)

Instead of applying computationally expensive 3D convolutions across all layers, the rMC-50 backbone uses 2D convolutions in the shallow layers (conv1 through conv4) to extract spatial features. 3D convolutions are reserved exclusively for the deep layers (conv5) to capture temporal dynamics. This "mixed" approach drastically reduces the number of parameters while maintaining the network's ability to understand complex motion.

Comparison of rMC and R3D architectures showing the transition from 2D to 3D convolutions.

RPN Integration and Target Localization

A Region Proposal Network (RPN) is coupled with the 2D layers of the backbone. It identifies the most likely location of the target and generates a "proposal box." By sharing the shallow 2D layers for both target detection and the initial stage of action recognition, the model avoids redundant calculations, contributing to its high-speed performance.

The RPN network architecture showing the flow from feature maps to proposal generation.

Crop Pooling and Refinement

The system utilizes the RPN’s highest-confidence proposal to "crop" the feature maps generated by the 2D layers. These target-specific features are then fed into the 3D (conv5) layers. This mechanism ensures that the temporal analysis is focused solely on the person of interest, effectively ignoring background noise. To further enhance precision, a regression block is placed between the crop pool and the final 3D layers to refine the bounding boxes, correcting initial detection inaccuracies.

Experimental Evaluation

The model was evaluated across three distinct datasets to validate its robustness and efficiency:

  1. UCF-101: A standard action recognition benchmark used to compare the rMC backbone against traditional R3D models.
  2. Custom Gesture Dataset: A self-manufactured dataset featuring 10 types of hand gestures (e.g., "Fist," "Right Swing") across 60 videos with varying backgrounds.
  3. Custom Body Posture Dataset: Designed to test the model's ability to ignore interference, this dataset features activities like "Eating" and "Reading" in environments with multiple people.

Key Results and Insights

Performance and Speed

On the UCF-101 dataset, the rMC-50 architecture achieved a 71.07% accuracy rate, notably higher than the standard R3D model's 66.65%. In terms of speed, the base model is exceptionally fast, reaching 200 FPS. Even with the addition of the accuracy-boosting regression block, it maintains a respectable 117 FPS, making it highly suitable for real-time edge applications.

Target Specificity

The research demonstrates that "wrong detections" in standard models are often caused by background actions. By cropping feature maps to the target, the model effectively "blinders" itself to distractions. On the custom Gesture Data, the improved model reached an Average Precision (AP) of 49.7% for detection and 70.7% for action recognition accuracy. For the Body Posture dataset, it achieved 64.8% recognition accuracy, proving its capability in handling larger-scale human movements amidst interference.

Conclusion

The rMC-RPN approach proves that 2D/3D fusion is a highly efficient strategy for action recognition. Reserving 3D operations for final temporal analysis saves significant compute while improving localization. While the current Average Precision for target detection remains an area for improvement, the system's ability to maintain high accuracy and real-time speeds in cluttered environments marks a significant step forward for practical action recognition systems. Future work will focus on refining target localization and scaling the model to more diverse, large-scale datasets.

Summary updated

Abstract

Abstract The traditional methods of action recognition are not specific for the operator, thus results are easy to be disturbed when other actions are operated in videos. The network based on mixed convolutional resnet and RPN is proposed in this paper. The rMC is tested in the data set of UCF-101 to compare with the method of R3D. The result shows that its correct rate reaches 71.07%. Meanwhile, the action recognition network is tested in our gesture and body posture data sets for specific target. The simulation achieves a good performance in which the running speed reaches 200 FPS. Finally, our model is improved by introducing the regression block and performs better, which shows the great potential of this model.
BibTeX
@misc{li2019action,
  author = {Mingjie Li and Youqian Feng and Zhonghai Yin and Cheng Zhou and Fanghao Dong and Lin Yuan and Yuhao Dong},
  title = {{An Action Recognition network for specific target based on rMC and RPN}},
  year = {2019},
  eprint = {1906.07944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07944},
}
Figure