Abstract
Action recognition in complex scenes remains challenging due to background interference and the presence of multiple moving objects. This paper introduces rMC-RPN, a hybrid architecture that combines target detection with action recognition to isolate and analyze specific individuals. By employing a "mixed" convolutional ResNet (rMC-50) that utilizes 2D convolutions for spatial features and 3D convolutions for temporal dynamics, the model achieves an optimal balance between efficiency and accuracy. Integrated with a Region Proposal Network (RPN) for precise target localization, the system reaches processing speeds of up to 200 FPS. Experimental results on the UCF-101 benchmark and custom datasets demonstrate that rMC-RPN significantly outperforms traditional 3D-CNN models in both speed and recognition accuracy.
Introduction
Most traditional action recognition models, such as C3D or Two-Stream networks, analyze entire video frames to classify activities. While effective in controlled environments with a single subject, these models often fail in "cluttered" real-world scenarios where multiple people or distracting moving objects are present. If a system cannot distinguish between the primary "operator" and background interference, its recognition accuracy typically plummets.
To address this, the authors propose treating action recognition as a composite task: first, identifying the specific target through detection, and subsequently analyzing their movements. The primary challenge lies in achieving this integration efficiently enough for real-time applications without the massive computational overhead typically associated with full 3D convolutional neural networks (3D-CNNs).
Proposed Methods
The core innovation of the rMC-RPN architecture is its strategic balance of spatial and temporal processing through a multi-stage pipeline.
Mixed Convolutional ResNet (rMC-50)
Instead of applying computationally expensive 3D convolutions across all layers, the rMC-50 backbone uses 2D convolutions in the shallow layers (conv1 through conv4) to extract spatial features. 3D convolutions are reserved exclusively for the deep layers (conv5) to capture temporal dynamics. This "mixed" approach drastically reduces the number of parameters while maintaining the network's ability to understand complex motion.
RPN Integration and Target Localization
A Region Proposal Network (RPN) is coupled with the 2D layers of the backbone. It identifies the most likely location of the target and generates a "proposal box." By sharing the shallow 2D layers for both target detection and the initial stage of action recognition, the model avoids redundant calculations, contributing to its high-speed performance.
Crop Pooling and Refinement
The system utilizes the RPN’s highest-confidence proposal to "crop" the feature maps generated by the 2D layers. These target-specific features are then fed into the 3D (conv5) layers. This mechanism ensures that the temporal analysis is focused solely on the person of interest, effectively ignoring background noise. To further enhance precision, a regression block is placed between the crop pool and the final 3D layers to refine the bounding boxes, correcting initial detection inaccuracies.
Experimental Evaluation
The model was evaluated across three distinct datasets to validate its robustness and efficiency:
- UCF-101: A standard action recognition benchmark used to compare the rMC backbone against traditional R3D models.
- Custom Gesture Dataset: A self-manufactured dataset featuring 10 types of hand gestures (e.g., "Fist," "Right Swing") across 60 videos with varying backgrounds.
- Custom Body Posture Dataset: Designed to test the model's ability to ignore interference, this dataset features activities like "Eating" and "Reading" in environments with multiple people.
Key Results and Insights
Performance and Speed
On the UCF-101 dataset, the rMC-50 architecture achieved a 71.07% accuracy rate, notably higher than the standard R3D model's 66.65%. In terms of speed, the base model is exceptionally fast, reaching 200 FPS. Even with the addition of the accuracy-boosting regression block, it maintains a respectable 117 FPS, making it highly suitable for real-time edge applications.
Target Specificity
The research demonstrates that "wrong detections" in standard models are often caused by background actions. By cropping feature maps to the target, the model effectively "blinders" itself to distractions. On the custom Gesture Data, the improved model reached an Average Precision (AP) of 49.7% for detection and 70.7% for action recognition accuracy. For the Body Posture dataset, it achieved 64.8% recognition accuracy, proving its capability in handling larger-scale human movements amidst interference.
Conclusion
The rMC-RPN approach proves that 2D/3D fusion is a highly efficient strategy for action recognition. Reserving 3D operations for final temporal analysis saves significant compute while improving localization. While the current Average Precision for target detection remains an area for improvement, the system's ability to maintain high accuracy and real-time speeds in cluttered environments marks a significant step forward for practical action recognition systems. Future work will focus on refining target localization and scaling the model to more diverse, large-scale datasets.
Summary updated