Improving Object Detection, Multi-object Tracking, and Re-Identification for Disaster Response Drones

Chongkeun Paik, Hyunwoo J. KimarXiv 2022 · OpenReview

TL;DR

This paper tackles the critical challenge of object detection, tracking, and re-identification for drones operating in disaster response scenarios. They propose two approaches that combine state-of-the-art models to improve accuracy and efficiency, demonstrating their effectiveness in real-world simulations and competitions.

What Problem Does This Paper Address?

Disaster response drones are increasingly vital for search and rescue, damage assessment, and situational awareness. However, effectively using computer vision on these drones presents significant challenges:

  • Detection Errors: Identifying objects accurately in complex, cluttered environments with varying lighting and partial occlusions.
  • ID Switching and Fragmentation: Maintaining a consistent identity for each detected object as it moves, especially when it's temporarily lost from view or when multiple similar objects are present.
  • Multi-Scale and Multi-View Issues: Handling objects that appear at different sizes and from various camera angles simultaneously.
  • Global Camera Motion: Dealing with the inherent instability and movement of drone-mounted cameras, which can disrupt tracking.
  • Resource Constraints: Operating within the limited computational power and time available on drone platforms.

The authors aim to develop robust and efficient systems that can overcome these hurdles for effective disaster response.

Methods

The paper proposes two main approaches, both leveraging existing powerful models but combining them in novel ways:

  • Approach 1: Fast Multi-Camera System with Tracklet Association: This approach focuses on improving tracking accuracy in crowded scenes with fast-moving individuals. It builds upon the FairMOT tracker by integrating OSNet for enhanced re-identification. OSNet is used to extract more robust feature vectors, which are then used to associate tracklets (sequences of detected object bounding boxes) across different camera views. This aims to reduce ID fragmentation and improve overall tracking consistency.
  • Approach 2: High-Performance Detector and Tracker for Irregular Motion: This approach targets real-world disaster scenarios with irregular camera movements and limited hardware. It replaces FairMOT with a combination of YOLOv5 (a high-performance object detector) and DeepSORT (a robust tracker). This combination is designed to enhance detection accuracy and handle challenging tracking conditions. The system also includes a post-processing step after the re-ID block to refine results.

Both approaches utilize OSNet for feature extraction to aid in tracklet association.

Datasets

  • Training:
    • CrowdedHuman
    • MOT17
    • MS-COCO (for object detection)
    • market1501 (for re-identification, pre-trained)
  • Validation:
    • A custom validation dataset consisting of 15 videos (5 sets of 3 videos each), with each video containing 300 frames. This dataset simulates disaster scenarios with fixed cameras from multiple views.
  • A custom validation dataset for Study-2 with 3 videos, each about 5 minutes long (approx. 7000 frames), featuring mannequins with partial body movements and stationary locations.
  • Test:
    • The test dataset for Study-1 comprised 500 sets of videos from the 2020 AI Grand Challenge.
  • The test dataset for Study-2 comprised 5 sets of videos from the 2021 AI Grand Challenge.
  • Note: The test datasets were not directly accessible to the challengers; only scores were provided.

Key Insights and Ideas

  • Synergy of Trackers and Re-ID Models: The paper demonstrates that combining specialized object trackers (like FairMOT or YOLOv5+DeepSORT) with powerful re-identification models (like OSNet) leads to significant improvements, especially in challenging multi-camera scenarios.
  • OSNet's Effectiveness for Re-ID: OSNet proves to be a valuable component for extracting discriminative features, crucial for accurately associating tracklets across different camera views and reducing ID switching.
  • Trade-offs in Real-World Applications: The choice of approach (FairMOT+OSNet vs. YOLOv5+DeepSORT) depends on the specific constraints. The first is optimized for speed and crowded scenes, while the second prioritizes detection accuracy for more dynamic and resource-constrained environments.
  • Importance of Post-Processing: For moving camera scenarios, a post-processing step is crucial to refine detections and remove false positives, enhancing overall system robustness.
  • Competition Success: The models' effectiveness is validated by their strong performance (second and third place) in the "AI Grand Challenge for Disaster Response Drones," highlighting their practical applicability.

Key Results

  • Study-1 (FairMOT + OSNet):
    • Achieved an accuracy of 85.71% on the validation dataset, a slight improvement over the baseline FairMOT (85.44%).
  • Significantly reduced L2-norm error from 48.1 (baseline) to 34.9, a 27.4% reduction.
  • Enabled processing at 27 FPS, close to real-time.
  • Study-2 (YOLOv5-DeepSORT + OSNet):
    • Outperformed FairMOT in terms of recall (71.4% vs. 42.9%), even though DeepSORT processed only a quarter of the frames due to hardware limitations.
  • Achieved 62.5% accuracy and 76.9% F1 score, compared to FairMOT+OSNet's 42.9% accuracy and 60% F1 score.
  • Qualitative Results: Visualizations (Figure 4.6) show that the proposed models are better at distinguishing individuals and assigning consistent IDs across different camera views compared to the baseline.

Limitations and Future Work

  • Limitations:
    • Resource Constraints: Study-2 faced significant hardware limitations, leading to processing only a fraction of frames, which still yielded competitive results.
  • Generalizability of Voting Method: The voting method for association showed better performance on small datasets but tended to degrade on larger test sets, indicating potential generalizability issues.
  • Distinguishing Non-Person Objects: Heuristic methods for distinguishing non-person objects by threshold have limitations and can be improved with deep learning and more data.
  • Future Work:
    • Investigate better metrics beyond traditional methodologies.
  • Improve the multi-camera system by combining various object trackers and re-ID solutions, considering insights from both studies.
  • Further enhance the output refinement block for greater overall performance impact.
  • Propose a one-stage Online Multi-Camera Tracking model for improved efficiency and speed.

Summary updated

Abstract

We aim to detect and identify multiple objects using multiple cameras and computer vision for disaster response drones. The major challenges are taming detection errors, resolving ID switching and fragmentation, adapting to multi-scale features and multiple views with global camera motion. Two simple approaches are proposed to solve these issues. One is a fast multi-camera system that added a tracklet association, and the other is incorporating a high-performance detector and tracker to resolve restrictions. (...) The accuracy of our first approach (85.71%) is slightly improved compared to our baseline, FairMOT (85.44%) in the validation dataset. In the final results calculated based on L2-norm error, the baseline was 48.1, while the proposed model combination was 34.9, which is a great reduction of error by a margin of 27.4%. In the second approach, although DeepSORT only processes a quarter of all frames due to hardware and time limitations, our model with DeepSORT (42.9%) outperforms FairMOT (71.4%) in terms of recall. Both of our models ranked second and third place in the `AI Grand Challenge' organized by the Korean Ministry of Science and ICT in 2020 and 2021, respectively. The source codes are publicly available at these URLs (github.com/mlvlab/drone_ai_challenge, github.com/mlvlab/Drone_Task1, github.com/mlvlab/Rony2_task3, github.com/mlvlab/Drone_task4).
BibTeX
@misc{paik2022improving,
  author = {Chongkeun Paik and Hyunwoo J. Kim},
  title = {{Improving Object Detection, Multi-object Tracking, and Re-Identification for Disaster Response Drones}},
  year = {2022},
  eprint = {2201.01494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.01494},
}
Figure