TL;DR
This paper introduces "Psych-ER," a novel human-guided dataset and loss function for improving aerial person detection in challenging search and rescue (SAR) scenarios. By analyzing how humans visually search for people in occluded and distant aerial images, they've developed a method to make computer vision models more robust to these real-world complexities, outperforming standard detection methods.
What Problem Does This Paper Address?
Emergency response (ER) scenarios, particularly search and rescue (SAR), heavily rely on quickly locating individuals. Small Unmanned Aerial Systems (sUAS) are increasingly used as "eyes in the sky" for this purpose. However, existing computer vision (CV) models struggle with the harsh realities of aerial SAR: low target resolution due to distance and significant occlusion (people being partially hidden by objects like trees, debris, or smoke). Human operators, while better at handling these challenges, suffer from fatigue and limited resources. This paper aims to bridge this gap by developing CV models that can better mimic human visual perception in these difficult conditions, specifically focusing on the critical task of locating a person, not just classifying them.
Methods
The core innovation lies in leveraging human visual behavior to improve CV models. The authors first created a dataset called Psych-ER by collecting behavioral data from human participants performing a "find the person" task on aerial images from the NOMAD dataset. This data includes not only whether the person was found (accuracy) but also how they searched (attention location, duration, and response time).
Using this behavioral data, they developed a psychophysical loss function for object detection models. Instead of solely relying on traditional bounding box regression losses (which focus on precise box fitting), their loss function penalizes predictions based on how well they align with human search patterns. Specifically, it incorporates a "human penalty" that is inversely proportional to the human performance ([email protected]) at different distances and visibility levels. This means the model is guided to prioritize finding the location of the person, mirroring human behavior, especially in challenging scenarios. They then integrated this loss function into a RetinaNet model.
Datasets
- NOMAD Dataset: This is the primary dataset used for extracting images. It contains aerial images of people at various distances (10m to 90m) and with different levels of occlusion (visibility labels). The authors previously introduced this dataset.
- Psych-ER Dataset: This is a newly introduced behavioral dataset derived from the NOMAD dataset. It comprises over 5,000 images collected from a crowdsourced experiment where participants searched for people. The dataset includes:
- Human search behavioral data (cursor location, area of attention, duration).
- Accuracy of human selections relative to NOMAD's ground truth bounding boxes.
- Response times for each image.
- A subset of "control" images for quality assurance.
Key Insights and Ideas
- Human search is location-centric: The analysis of human behavior in the Psych-ER dataset revealed that when asked to "find the person," humans prioritize locating the person's general area rather than precisely fitting a bounding box around them. This is a crucial insight for designing detection models for ER scenarios where recall is paramount.
- Occlusion and distance are major CV hurdles: The study confirms that both increasing distance and occlusion significantly degrade human performance, but CV models are even more susceptible to these challenges.
- Psychophysics can guide CV: By quantifying human visual performance under varying conditions, psychophysical data can be directly translated into loss functions that guide CV models to perform better in similar real-world scenarios.
- Focus on location over tightness: The proposed psychophysical loss function specifically targets the location component of bounding box regression, acknowledging that for ER, finding where the person is, is more critical than a perfectly tight bounding box.
Key Results
The RetinaNet model trained with the psychophysical loss function showed improvements over the baseline model (trained with default loss) in aerial person detection.
- Improved performance at higher distances: The psychophysical loss adaptation demonstrated significant improvements at greater distances (e.g., 70m) and across various occlusion levels.
- No degradation at closer distances: Crucially, the adapted loss did not negatively impact performance at closer distances where detection is generally easier.
- Minimal computational overhead: The proposed method adds negligible computational cost during inference, making it suitable for deployment on resource-constrained sUAS.
Limitations and Future Work
- Limitations: The paper acknowledges that occlusion remains a significant challenge, with performance dropping considerably for visibility scores below 50% and at greater distances. The current psychophysical loss is primarily based on human accuracy ([email protected]), and other behavioral data like response time and search patterns are not yet fully exploited.
- Future Work: The authors plan to:
- Exploit the full range of behavioral data (response time, search patterns) to further improve CV models across various architectures.
- Develop custom CV architectures specifically for ER scenarios, potentially incorporating sUAS flight data (e.g., altitude).
- Explore training CV models with the psychophysical loss for specific altitudes, optimizing onboard CV pipelines for sUAS.
- Ultimately, deploy and evaluate these ER-tailored CV models on physical sUAS in real-world conditions.
- Investigate analyzing contiguous image frames to better handle dynamic scenarios and adapt sUAS flight paths based on detection confidence.
Summary updated