Detecting Tear Gas Canisters With Limited Training Data

Ashwin D'Cruz, Christopher Tegho, Sean Greaves, Lachlan KermodeWACV 2022

TL;DR

This paper tackles the challenge of automatically detecting tear gas canisters in images for human rights investigations, a task hindered by a lack of sufficient training data. They introduce a new dataset and explore various object detection techniques, including fine-tuning pre-trained models, few-shot learning, and synthetic data augmentation, to achieve effective detection with limited real-world examples.

What Problem Does This Paper Address?

Human rights investigations often rely on sifting through vast amounts of open-source media to identify crucial moments. Manually searching for specific objects like tear gas canisters is incredibly time-consuming and labor-intensive. While object detection models, particularly those based on Convolutional Neural Networks (CNNs), are powerful tools for automation, they typically require large, annotated datasets for training. Unfortunately, a comprehensive dataset for tear gas canisters simply doesn't exist, creating a significant bottleneck for developing automated detection systems in this critical domain. This paper aims to bridge this gap by investigating how to train effective tear gas canister detectors with limited available real-world data.

Methods

The authors explore several strategies to overcome the limited training data challenge:

  • Fine-tuning Pre-trained Models: They leverage state-of-the-art object detection models (YOLOv5, Faster R-CNN, and RetinaNet) that have been pre-trained on large, general-purpose datasets like COCO. These models are then fine-tuned on their newly created, smaller dataset of tear gas canisters.
  • Synthetic Data Augmentation: To artificially increase the training data, they generate synthetic images of tear gas canisters using 3D modeling software (Unity and Unreal Engine). These synthetic images are designed to mimic real-world visual contexts and variations.
  • Few-Shot Learning: They investigate few-shot learning techniques, specifically a method involving feature reweighting, which is designed to learn from a very small number of labeled examples. This approach aims to adapt models to new classes with minimal training data.
  • "Similar" Object Classes: For the few-shot detector, they also explore the impact of including images of objects that resemble tear gas canisters (e.g., bottles, cans) in the training set to improve generalization.

Datasets

The paper introduces a new, publicly available dataset for tear gas canister detection. This dataset comprises four collections:

  1. Real Images with Canisters: Images containing one or more tear gas canisters, varying in type, view angle, size, color, and state, captured in diverse real-world scenes.
  2. "Similar" Objects: Images of objects that visually resemble tear gas canisters but are not.
  3. Scenes without Canisters: Images of scenes where canisters have appeared but are not currently present.
  4. Synthetic Images: Artificially generated images of tear gas canisters to augment the training set.

The dataset is split into training, validation, and testing sets, with specific counts for each subset provided in Table 1.

Key Insights and Ideas

  • Synthetic data can help, but real data is king: While synthetic data augmentation can improve performance for some models, the paper highlights that real image data, even in small quantities, is often more crucial for achieving high mAP (mean Average Precision).
  • Fine-tuning pre-trained models is effective: Leveraging models pre-trained on large datasets like COCO provides a strong foundation, and fine-tuning them on even a limited set of real tear gas canister images yields competitive results.
  • "Similar" object classes are vital for few-shot learning: For few-shot detectors, including images of objects that look like tear gas canisters in the training set significantly improves performance, suggesting that learning to differentiate from visually similar but incorrect objects is key.
  • Model capacity matters: For YOLOv5, larger models generally achieve better performance, indicating that increased model complexity can better capture the nuances of the task when sufficient data (real or synthetic) is available.
  • F2 score is a more relevant metric for this application: The authors emphasize that for human rights investigations, minimizing missed detections (recall) is often more critical than perfect bounding box accuracy. The F2 score, which balances recall and reduction (false positives), is presented as a more suitable evaluation metric.

Key Results

The paper presents extensive results across various methods and configurations. Here are some highlights:

  • Faster R-CNN and RetinaNet: These two-stage and single-stage detectors, respectively, achieved the highest mAP scores on the test set, with Faster R-CNN reaching up to 66.73% mAP when trained on real data only.
  • YOLOv5: The large YOLOv5 model achieved competitive mAP scores, around 19.21% with real and synthetic data. While not as high as Faster R-CNN, YOLOv5 offers significantly faster inference speeds.
  • Few-Shot Detector: The few-shot detector showed promising results, especially when trained with a small number of real images (10) combined with synthetic and "similar" object images, achieving an F2 score of 0.7594 on the validation set.
  • Impact of Synthetic Data: For Faster R-CNN and RetinaNet, using synthetic data sometimes resulted in worse performance compared to using real data alone. However, for the few-shot detector, synthetic data generally improved results.

Limitations and Future Work

  • Limitations:
    • The dataset, while novel, is still relatively small compared to standard object detection benchmarks.
  • The performance of some models, particularly YOLOv5, did not show a clear trend with the F2 score, possibly due to optimization primarily for mAP.
  • The validation set included "similar" objects, which might have influenced results compared to a test set containing only actual tear gas canisters.
  • Future Work:
    • Improved Optimization Metrics: Incorporating metrics like recall and F2 directly into the optimization process to better align with user needs.
  • Detailed Synthetic Data Analysis: A deeper investigation into the impact of scene complexity and resolution in synthetic data generation.
  • Real-World Deployment Analysis: Evaluating the detectors' utility for various actors (research organizations, citizen data scientists, activists) in real-world deployment scenarios.
  • Exploring Other Augmentation Techniques: Further research into data augmentation strategies tailored for this specific domain.

Summary updated

BibTeX
@inproceedings{dcruz2022detecting,
  author = {Ashwin D'Cruz and Christopher Tegho and Sean Greaves and Lachlan Kermode},
  title = {{Detecting Tear Gas Canisters With Limited Training Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure