TL;DR
This paper introduces WEPDTOF, the first "in-the-wild" dataset for people detection and tracking from overhead fisheye cameras, addressing the limitations of existing "staged" datasets. They also propose three spatio-temporal extensions to a state-of-the-art algorithm, demonstrating significant performance improvements on their new benchmark.
What Problem Does This Paper Address?
Traditional people detection and tracking algorithms, often developed for standard rectilinear cameras, struggle with the unique geometry and distortions introduced by overhead fisheye (OF) cameras. While recent progress has been made, existing research has been hampered by a lack of diverse and realistic datasets. Current benchmarks consist of "staged" recordings with limited scenarios, people, and challenges, making it unclear how these algorithms would perform in real-world surveillance applications. This paper aims to bridge this gap by providing a more challenging dataset and evaluating algorithms under more realistic conditions.
Methods
The authors introduce three novel spatio-temporal extensions to RAPiD, a state-of-the-art fisheye people detection algorithm. These extensions leverage temporal information, which has shown promise in standard video object detection.
- RAPiD+REPP: This approach applies REPP (Robust and Efficient Post-Processing) to the bounding boxes generated by RAPiD. REPP uses a learning-based similarity function to link bounding boxes across consecutive frames, creating "tubelets" and then refining them to improve confidence scores, locations, and sizes.
- RAPiD+FA: This method integrates feature aggregation without explicit feature warping. It aggregates feature maps from consecutive frames using adaptive weights.
- RAPiD+FGFA: This is an end-to-end approach that leverages optical flow to warp feature maps from past and future frames to the current frame before aggregation. This allows for a more comprehensive integration of temporal information.
For tracking, they propose a simple similarity-based association algorithm applied to the bounding boxes generated by their detection algorithms, using the same learned similarity function as in RAPiD+REPP.
Datasets
- Training Data: A combination of existing datasets (MS COCO, MW-R, HABBOF, and CEPDOF) was used for training.
- Evaluation Dataset: The paper introduces WEPDTOF (in-the-Wild Events for People Detection and Tracking from Overhead Fisheye cameras). This dataset consists of 14 YouTube videos (resulting in 16 clips) captured in diverse "in-the-wild" scenes, featuring 188 distinct person identities, and includes real-world challenges like severe occlusions and camouflage. It is significantly larger and more diverse than previous fisheye datasets.
Key Insights and Ideas
- "In-the-Wild" is Crucial: The performance of people detection algorithms on staged datasets doesn't translate well to real-world scenarios. WEPDTOF's diverse, unscripted nature is essential for truly evaluating robustness.
- Temporal Information Boosts Performance: Integrating temporal information significantly improves people detection accuracy, even for algorithms that were already strong in spatial detection.
- Challenges of Fisheye Tracking: People tracking in fisheye images is a nascent area. The proposed tracking method shows promise but highlights the need for more advanced tracking techniques adapted for fisheye geometry.
- Small Objects are a Major Hurdle: The paper notes a significant performance drop for detecting small people, indicating this is a key area for future research.
Key Results
- WEPDTOF Dataset: Contains 16 video clips, 188 unique identities, and challenges like occlusions and camouflage.
- Detection Performance: The proposed spatio-temporal extensions to RAPiD (RAPiD+REPP, RAPiD+FA, RAPiD+FGFA) significantly outperform baseline algorithms on WEPDTOF. RAPiD+FGFA achieved the highest AP50 score of 76.6%.
- Tracking Performance: RAPiD+FA and RAPiD+FGFA showed improved tracking performance (e.g., MOTA of 47.7% and IDF1 of 53.3% for RAPiD+FA) compared to RAPiD alone.
- Efficiency: While spatio-temporal methods improve accuracy, they come with increased computational cost. RAPiD+FGFA is significantly slower than the original RAPiD.
Limitations and Future Work
- Dataset Size for Training: WEPDTOF is primarily used for evaluation due to its diversity, but a larger, annotated "in-the-wild" dataset specifically for training would be beneficial.
- Tracking Sophistication: The current tracking approach is a simple association method. Future work could explore adapting advanced multi-object tracking (MOT) techniques from standard camera domains to fisheye cameras.
- Small Object Detection: Performance on small objects remains a significant challenge, and this is identified as an open research direction.
- Computational Efficiency: The spatio-temporal algorithms are not yet real-time. Improving efficiency is crucial for practical deployment.
Summary updated