TL;DR
This paper tackles the challenge of recognizing multiple people's actions in 360° videos, a growing area due to affordable omnidirectional cameras. They introduce a novel framework that first "unwarps" the distorted 360° footage into a more standard panoramic view and then employs a weakly-supervised, multi-instance learning approach to identify actions using only video-level labels.
What Problem Does This Paper Address?
Omnidirectional cameras, like those found in many surveillance systems, offer a wide field of view, capturing an entire scene with a single device. This is cost-effective and simplifies setup compared to multiple conventional cameras. However, analyzing this 360° video data presents unique challenges for existing action recognition models. Firstly, the extreme distortion in 360° footage causes people to appear deformed, making it difficult for standard deep learning models, trained on perspective views, to extract useful features. Secondly, these cameras often capture multiple people performing actions simultaneously, and annotating each individual's actions precisely is extremely time-consuming and expensive. This paper aims to bridge the gap between the increasing availability of 360° video hardware and the limited progress in analyzing such data, specifically for multi-person action recognition.
Methods
The proposed framework consists of two main stages:
- Omnidirectional to Panoramic Transformation: To address the distortion issue, the authors propose a calibration-free method to transform the top-view omnidirectional video into a panoramic view. This process involves identifying the center of the fisheye image (where lines perpendicular to the ground intersect) and then establishing a mapping from panoramic pixels to fisheye pixels. This transformation effectively "unwarps" people, making them appear upright and suitable for standard CNN processing.
- Weakly-Supervised Multi-Instance Multi-Label (MIML) Learning: For action recognition, the authors leverage a 3D Convolutional Neural Network (CNN), specifically a pre-trained 3D ResNet, to extract spatial-temporal features from clips of the panoramic video. To handle the multi-person, multi-action scenario with only video-level labels, they employ a MIML approach.
- Person Detection and Masking: They first use a person detector (Mask-RCNN) to identify people in the video frames. This information is used to create a binary mask that highlights regions containing people, effectively filtering out background noise and focusing the CNN on relevant areas.
- Instance Division: The masked feature maps are then divided into multiple "instances" (blocks).
- Action Score Aggregation: Each instance's features are processed to generate action scores. Instead of simple averaging or max-pooling across instances, they use a Log-Sum-Exp (LSE) function to aggregate these scores. LSE provides a smooth way to combine instance scores, allowing for a balance between considering the strongest individual instance and the collective evidence from multiple instances.
- Sparsity Regularization: A regularization term is introduced to penalize instances that output high scores for multiple actions, encouraging each instance to be confident about a single action.
Datasets
- 360Action Dataset: The authors introduce a new dataset called "360Action," which is claimed to be the first omnidirectional video dataset for multi-person action recognition. It features 784 high-resolution (2880x2880) videos captured by a Kodak PixPro SP360 4K camera. The dataset covers 10 diverse scenes (indoors and outdoors) and includes 19 classes of actions, with multiple people performing actions concurrently in each video. Subjects are split into training (IDs 1-60) and testing (IDs 61-80) sets.
Key Insights and Ideas
- Unwarping for 360° Video: The simple, calibration-free transformation of distorted 360° video into a panoramic view is a crucial preprocessing step that makes existing CNN architectures applicable.
- Weak Supervision is Key: The ability to train a robust action recognition model using only video-level labels (without precise bounding boxes or action localization for each person) significantly reduces annotation costs and makes large-scale training feasible.
- MIML for Multi-Person Actions: The Multi-Instance Multi-Label learning framework, particularly with the Log-Sum-Exp aggregation and sparsity regularization, effectively handles the complexity of multiple people performing actions simultaneously in a single video.
- Sparsity Encourages Specificity: The sparsity regularization term is vital for ensuring that individual instances contribute to recognizing a specific action, rather than being overly confident about multiple actions.
Key Results
- State-of-the-Art Performance: The proposed MIML framework achieved a mean average precision (mAP) of 70.12% on the 360Action dataset, significantly outperforming several state-of-the-art baselines (e.g., Collective, 3D ResNet, R-C3D, MiCT) which were adapted for this task.
- Component Efficacy: Ablation studies demonstrated the importance of each component:
- Using the region mask improved performance by +2.5%.
- Sparsity regularization further improved performance by +0.87%.
- The LSE aggregation method outperformed attention pooling, max, and average pooling.
- Action Localization: The model was able to localize predicted actions by generating heatmaps, demonstrating that it learns to associate actions with specific people even without explicit location supervision.
- Inference Speed: While training is efficient, inference speed is limited by video resolution. Using a shallower 3D ResNet-18 and a single-stage detector (YOLO) achieved 6.28 FPS, a trade-off for lower recognition performance.
Limitations and Future Work
- Inference Speed: The high resolution of 360° videos poses a challenge for real-time inference. The authors suggest using shallower feature extractors and single-stage detectors to improve speed, but this comes at the cost of accuracy.
- Action Localization Granularity: While qualitative results show localization, the paper doesn't detail quantitative metrics for localization accuracy.
- Future Directions: The authors hope their work will encourage further research in omnidirectional video analysis, aiming to achieve the same level of success as in conventional video analysis. This could include exploring more efficient architectures for high-resolution videos, improving localization accuracy, and expanding the range of actions and scenarios.
Summary updated