Defending Object Detection Models Against Image Distortions

Mark Ofori-Oduro, Maria AmerWACV 2024

TL;DR

This paper introduces GSES, a novel data augmentation technique that generates new training samples by intelligently distorting existing ones. By sampling pixel values from distributions derived from multiple real-world distortions, GSES enhances object detection models' robustness to various image corruptions without sacrificing performance on clean images.

What Problem Does This Paper Address?

Object detection models are crucial for many real-world applications, from autonomous driving to security systems. However, their performance significantly degrades when faced with image distortions like noise, blur, or compression artifacts. While simply adding distortions to training data can improve robustness, it often leads to a drop in accuracy on clean images and poor generalization to unseen distortions. Existing data augmentation methods and defense strategies have limitations, such as complexity, reduced performance on clean data, or a lack of generalization. This paper aims to bridge this gap by proposing a data augmentation method that effectively defends object detection models against a wide range of image distortions while maintaining high accuracy on clean images.

Methods

The core of the proposed method, GSES (Generative Sample Enhancement Strategy), involves a novel data augmentation approach. It works by:

  1. Data Pre-processing: Randomly selecting a subset of original training images to generate new, distorted samples.
  2. New Sample Generation: For each selected image, GSES first generates multiple distorted versions using a set of 15 predefined distortion types (e.g., Gaussian noise, motion blur, fog). Then, for a randomly selected fraction of pixels in the original image, it estimates the distribution of pixel values from the corresponding pixels across these distorted versions using Kernel Density Estimation (KDE). Finally, it samples a new pixel value from this estimated distribution and replaces the original pixel. This process ensures that the generated samples retain a degree of similarity to the original image, balancing robustness with clean image accuracy.
  3. Data Augmentation: The newly generated samples are then used to augment the original training dataset.

A key aspect of GSES is its non-parametric approach using KDE, which doesn't assume any specific distribution for the distortions. It also strategically mutates only a fraction of pixels, preserving some original information and preventing overfitting.

Datasets

The paper evaluates its method on two widely used object detection datasets:

  • PASCAL VOC: Used for training and validation.
  • COCO: Used for training and validation.

The authors also mention evaluating on autonomous driving datasets like KITTI and BDD100K, as well as image classification datasets like CIFAR-10, CIFAR-100, and Caltech-101 to demonstrate cross-domain applicability.

Key Insights and Ideas

  • Pixel-level distribution sampling is key: Instead of applying global distortions, sampling pixel values from estimated distributions of various distortions allows for more nuanced and effective augmentation.
  • Balancing similarity and distortion is crucial: Mutating only a fraction of pixels (controlled by the affinity threshold $\tau$) is vital for maintaining a balance between performance on clean and distorted images. Too much mutation can hurt clean image accuracy.
  • Diversity of distortions matters: Using a diverse set of 15 distortions for KDE estimation leads to better generalization and robustness against unknown distortions.
  • No need for complex architectures: GSES is a data augmentation technique that doesn't require modifying the object detection model's architecture, making it easily applicable to existing models.

Key Results

  • Significant accuracy improvements: GSES consistently improved the mean Average Precision (mAP) for YOLOv4 and Faster RCNN models across 15 image distortions. For YOLOv4 on PASCAL, it achieved a +9.54% increase in mPC (mean performance under corruption).
  • No degradation on clean images: Unlike some other methods, GSES did not lead to a drop in accuracy on clean images.
  • Generalization to unknown distortions: The method demonstrated strong performance on distortions not explicitly used during training, outperforming other defense methods.
  • Applicability to image classification: GSES also showed benefits when applied to image classification tasks using the SpinalNet model.
  • Competitive computational cost: While generating new samples takes time, it's presented as a reasonable trade-off for the significant performance gains.

Limitations and Future Work

  • Hyperparameter tuning: The method relies on hyper-parameters like the augmentation ratio ($p$) and affinity threshold ($\tau$), which require careful tuning for optimal performance.
  • Computational cost of KDE: While manageable, the KDE estimation and sampling process adds to the training time. The paper notes that blur distortions are more computationally intensive.
  • Future work: The authors suggest exploring more sophisticated KDE kernels and investigating the impact of different distortion sets. They also hint at further research into cross-domain generalization and applying the method to other computer vision tasks.

Summary updated

BibTeX
@inproceedings{oforioduro2024defending,
  author = {Mark Ofori-Oduro and Maria Amer},
  title = {{Defending Object Detection Models Against Image Distortions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure