TL;DR
This paper introduces EDGE20, a novel cross-spectral (VIS/NIR) dataset for surveillance tasks like pedestrian and face detection/recognition, captured under unconstrained outdoor conditions. They also provide baseline performance results for state-of-the-art methods, highlighting the significant challenges posed by this realistic, unconstrained, and cross-spectral setting.
What Problem Does This Paper Address?
Current surveillance datasets often focus on a single task (e.g., just pedestrian detection) and are typically captured using visible light (VIS) cameras. While some cross-spectral datasets exist, they are often collected in controlled environments, limiting their applicability to real-world surveillance scenarios. This paper addresses the need for a dataset that:
- Covers multiple surveillance problems: Pedestrian detection, face detection, and face recognition.
- Utilizes cross-spectral imaging: Specifically, visible (VIS) and near-infrared (NIR) spectra.
- Captures data in unconstrained environments: Mimicking real-world surveillance with variations in lighting (day/night), pose, resolution, occlusion, and motion blur.
Existing datasets for face recognition, for instance, are often limited to well-lit indoor settings, making them insufficient for evaluating systems intended for outdoor surveillance. EDGE20 aims to bridge this gap by providing a challenging benchmark for these critical tasks.
Methods
The core contribution of this paper is the EDGE20 dataset itself. The data acquisition involved:
- Trail Cameras: Four NIR-VIS trail cameras were strategically placed to cover a bi-directional trail path.
- Unconstrained Outdoor Environment: Data was collected in an outdoor setting, mimicking a trail path, with cameras positioned at varying heights and angles.
- Day and Night Sessions: Images were captured during both day and night to account for illumination variations.
- Subject Variety: 197 subjects participated, with some wearing accessories (hats, backpacks) to simulate occlusions.
- Rich Annotations: The dataset includes person and facial bounding boxes, unique subject IDs, and labels for facial poses (frontal, profile, back).
To establish baseline performance, the authors evaluated several state-of-the-art methods:
- Pedestrian Detection: Faster R-CNN [22] was used.
- Face Detection: SANet [24] was employed.
- Face Recognition: ArcFace [6] was utilized.
The evaluation protocols are clearly defined, covering various combinations of VIS and NIR spectra for different tasks, and are performed with ground truth from two annotators to account for potential bias.
Datasets
- EDGE20 Dataset (New Benchmark): This is the primary contribution. It comprises VIS and NIR images captured from trail cameras under unconstrained outdoor conditions, day and night. It includes annotations for pedestrian and face detection, as well as subject IDs for face recognition.
- Existing Datasets (for training/comparison):
- COCO dataset [19]: Used for pre-training the Faster R-CNN model.
- WIDER Face dataset [30]: Used for training and evaluating the SANet face detector.
- Deep Glint dataset [5]: Used for training the ArcFace face recognition model.
- CASIA NIR-VIS 2.0 dataset [18]: Used for evaluating the trained ArcFace model for NIR-VIS face recognition.
Key Insights and Ideas
- Cross-spectral challenges are significant: The results clearly demonstrate that current state-of-the-art methods struggle considerably when transitioning between VIS and NIR spectra, especially under unconstrained conditions.
- Unconstrained environments are a major hurdle: Factors like low resolution, pose variation, illumination changes, occlusions, and motion blur, common in real-world surveillance, severely degrade performance across all tasks.
- NIR presents unique difficulties: Detecting and recognizing faces/pedestrians in NIR images, particularly at night, is shown to be more challenging than in VIS images.
- Profile faces are particularly difficult: Both face detection and recognition performance drops significantly when dealing with profile views, a common scenario in surveillance.
- Dataset design matters: The EDGE20 dataset's design, incorporating real-world challenges, is crucial for pushing the boundaries of surveillance technology.
Key Results
- Pedestrian Detection: Faster R-CNN achieved higher performance on VIS images (PD1 protocol) compared to NIR images (PD2 protocol), with a performance gap of 1.4% to 1.5% across annotators. Qualitative results highlight challenges like low resolution and motion blur in NIR images.
- Face Detection: SANet showed a lower detection rate for profile faces compared to frontal faces, with a gap of 15.44% to 17.83%. The mAP scores for SANet on EDGE20 were around 69.58% and 70.49%.
- Face Recognition: ArcFace, while achieving state-of-the-art on other datasets, faced significant performance drops on EDGE20, especially for profile faces. For VIS-VIS recognition, performance dropped by over 55% when switching from frontal to profile probes. For NIR-VIS and NIR-NIR, the performance gap was at least 25%. The lowest rank-1 identification rates were observed for protocols involving profile faces in NIR settings (FR4 and FR8).
Limitations and Future Work
- Limitations:
- The dataset, while comprehensive, is still relatively small in terms of the number of unique subjects (197) and total images (3,494) compared to some massive datasets.
- The authors acknowledge that some subjects were manually blurred to preserve privacy, which might affect certain analyses.
- The evaluation of face recognition protocols FR4 and FR8 (profile faces in NIR) showed very low performance, indicating a significant area for improvement.
- Future Work:
- The paper sets the stage for future research by providing a challenging benchmark. The authors implicitly suggest that future work should focus on developing more robust algorithms capable of handling the unconstrained and cross-spectral challenges presented by EDGE20.
- Specifically, improving the detection and recognition of profile faces in NIR, and mitigating the effects of low resolution, occlusion, and motion blur are key areas for future research.
Summary updated