TL;DR
This paper introduces NOMAD, a novel aerial dataset specifically designed to address the critical challenge of human detection under occlusion in emergency response scenarios. By providing diverse, multi-scale aerial imagery with detailed visibility annotations, NOMAD aims to significantly improve the performance of computer vision systems used by drones in critical situations like search and rescue.
What Problem Does This Paper Address?
The increasing use of Unmanned Aerial Systems (sUAS) in emergency response, such as search and rescue, has highlighted a significant gap in computer vision capabilities: detecting humans from aerial views, especially when they are partially or fully occluded. While existing aerial datasets exist, none specifically focus on the pervasive issue of occlusion, which is a common and critical challenge in real-world emergency scenarios (e.g., victims hidden by debris, smoke, or natural elements). This lack of specialized data hinders the development of robust CV models that can reliably assist first responders, potentially making the difference between life and death. Furthermore, aerial views introduce unique challenges like varying distances, atmospheric conditions, and motion blur, which exacerbate the occlusion problem.
Methods
The paper introduces the Natural Occluded Multi-scale Aerial Dataset (NOMAD). The dataset was collected using a controlled process involving 100 actors performing various routines (walking, laying, hiding) in diverse natural and man-made locations. Key methodological aspects include:
- Multi-scale Data Collection: Videos were captured from five distinct altitudes (10m, 30m, 50m, 70m, and 90m) to simulate varying aerial perspectives and resolutions.
- Controlled Occlusion: Actors were instructed to hide behind obstacles, and routines were designed to naturally introduce occlusion.
- Detailed Annotation: Each of the 42,825 annotated frames includes a bounding box for each actor and a visibility label. This label quantifies the percentage of the human body visible within the bounding box, categorized into 10 distinct levels, providing a fine-grained measure of occlusion.
- Natural Variance: The dataset incorporates natural variations in filming locations, cross-seasonal imagery, and demographic diversity (age, gender, race) to enhance model robustness.
- Baseline Evaluation: The authors evaluated three state-of-the-art object detection models (YOLOv8l, FasterRCNN-R101-FPN, and RetinaNet-R101-FPN) on NOMAD to demonstrate its utility and highlight the challenges posed by occlusion and distance.
Datasets
The primary contribution is the NOMAD dataset itself.
- NOMAD: Contains 500 routine videos from 100 actors, totaling 42,825 frames. It features multi-scale aerial views (10m to 90m altitude) with 10 levels of occlusion visibility annotation. The dataset also includes demographic and environmental metadata.
- Existing Datasets (for comparison): The paper reviews existing aerial datasets like HERIDAL, SARD, WiSARD, BIRDSAI, VisDrone, and UAVDT, highlighting their limitations in addressing occlusion and providing rich metadata for emergency response scenarios. Datasets like BRIAR, MEVID, UAV-Human, and P-DESTRE are noted for rich metadata but lack occlusion focus.
Key Insights and Ideas
- Occlusion is a critical, under-addressed challenge: The paper strongly emphasizes that occlusion is a major bottleneck for aerial CV in emergency response, and existing datasets do not adequately capture this.
- Multi-scale and occlusion are intertwined: The performance degradation observed in models increases significantly with both distance and occlusion, suggesting that models need to be robust to both simultaneously.
- Ground-truth data matters for aerial CV: The baseline results show that models trained primarily on ground-level data (like COCO) struggle with aerial views, especially under occlusion, underscoring the need for specialized datasets like NOMAD.
- Fine-grained visibility labels are crucial: The 10-level visibility annotation allows for a nuanced evaluation of detection performance across a spectrum of occlusion, enabling more targeted model development.
Key Results
The baseline evaluation using state-of-the-art models revealed:
- Significant performance degradation: Mean Average Precision (mAP) scores dropped drastically as occlusion levels increased and distances grew.
- Distance is a major factor: Models performed best at the closest distance (10m), likely due to similarity with ground-based training data, but still suffered from occlusion.
- YOLOv8l showed promising results: It generally outperformed FasterRCNN and RetinaNet on the NOMAD dataset, particularly at closer distances.
- NOMAD's utility: The results demonstrate that NOMAD is a challenging benchmark, effectively highlighting the limitations of current models in handling occluded aerial views.
Limitations and Future Work
- Limitations:
- The authors acknowledge that the non-rigid nature of actor routines introduces some uncertainty in the precise visibility label.
- While efforts were made to avoid non-participants, a few instances required video trimming or blacking out, which could slightly impact data integrity in those specific frames.
- Future Work:
- Improving CV models: Future work will focus on developing CV models that exploit human psychophysical metrics and temporal information to better handle occlusion.
- Person Re-identification: Enhancing person re-identification capabilities by leveraging software architectures for hybrid onboard/offboard solutions and integrating humans into the loop.
- Real-world Deployment: Deploying and evaluating occlusion-ready CV models on physical sUAS to assess performance in real-world conditions.
- Tracking: NOMAD's keyframe structure is suitable for assessing tracking techniques.
Summary updated