Rethinking Image Restoration for Object Detection

Shangquan Sun, Wenqi Ren, Tao Wang, Xiaochun CaoNeurIPS 2022 · OpenReview

TL;DR

Standard image restoration methods (like dehazing or low-light enhancement) are designed for human eyes, but they often confuse AI object detectors. This paper proposes a novel fine-tuning pipeline that uses a targeted adversarial attack to create "detector-friendly" pseudo ground truth images, allowing restoration models to improve downstream detection accuracy without sacrificing visual quality.

What Problem Does This Paper Address?

In the real world, autonomous vehicles and surveillance systems must "see" through haze, rain, and low light. Naturally, researchers use image restoration as a pre-processing step to clean up these images. However, there is a frustrating paradox: restored images that look great to humans often lead to worse performance for object detectors.

This happens because restoration models can introduce subtle artifacts or "covariate shifts"—statistical patterns that the detector wasn't trained on. Existing solutions usually involve "joint training" (training the restorer and detector together) or "domain adaptation." Unfortunately, these often result in messy, noisy images or cause the detector to perform poorly on normal, clean images. The authors ask: Can we make restoration models produce images that are both visually pleasing and optimized for machine vision?

Methods

The authors propose a "plug-and-play" fine-tuning pipeline that treats the restoration task as a multi-task optimization problem. Their key innovation is the generation of Pseudo Ground Truth (GT).

  1. Targeted Adversarial Attack: Instead of forcing a restoration model to match a perfectly clean image, the authors use a targeted adversarial attack to find a version of the clean image that is "detector-friendly." They perturb the clean image just enough so that the detector (e.g., YOLOv3) produces the most accurate possible predictions.
  2. ADAM-variant Attack Algorithm: To find these pseudo GTs efficiently, they developed a momentum-based ADAM-like iterative algorithm. Unlike standard attacks that use simple gradients, this method uses clipped and signed gradients with momentum estimation to navigate the detector's loss landscape more effectively.
Conceptual illustration of generating detector-friendly pseudo ground truth via targeted adversarial attacks and the resulting feature space shift.
  1. Fine-tuning: Once they have these "detector-optimized" pseudo GT images, they use them as the targets to fine-tune existing restoration models (like MSBDN for dehazing or ZeroDCE for low light).

The beauty of this approach is that the detector remains "frozen" during the process, ensuring its performance on high-quality images is never compromised.

Datasets

The researchers evaluated their method on several challenging benchmarks:

  • Dehazing: RTTS (a real-world hazy dataset with object detection annotations) and VOC_fog (a synthetic hazy dataset created from PASCAL VOC).
  • Low-light Enhancement: ExDark (images captured in naturally low-light environments) and VOC_dark (a synthetic low-light version of PASCAL VOC).

Key Insights and Ideas

  • Restoration is Ill-Posed: There isn't just one "correct" restored version of a blurry or hazy image. By using adversarial attacks, the authors essentially "pick" the specific version of a sharp image that aligns best with the detector's internal features.
  • Feature Shift Visualization: The paper shows that their "attacked" pseudo GTs shift the image features in the latent space closer to the space where the detector is most confident and accurate.
  • Decoupling Tasks: By decomposing the problem into "attack generation" and "restoration training," they avoid the stability issues common in joint training or GAN-based multi-task learning.

Key Results

The method consistently outperformed conventional training and state-of-the-art domain adaptation techniques:

  • Detection Boost: On the RTTS hazy dataset, using their method with the MSBDN restorer boosted YOLOv3's mAP (mean Average Precision) from 42.77% (no restoration) to 44.10%. Conventional restoration actually performed worse than doing nothing in some cases.
  • Visual Fidelity: Unlike previous multi-task methods which often produce "unnatural" images with artifacts, this method maintains high PSNR and SSIM scores, meaning the images still look excellent to human observers.
Qualitative comparison of object detection performance showing the superiority of the proposed method over baselines in low-light and hazy conditions.
  • Generalization: The pipeline proved effective across different architectures, including one-stage (YOLOv3) and two-stage (Faster-RCNN) detectors.

Limitations and Future Work

  • Learning Capacity: The authors note that while a "perfect" detector-friendly image might exist, the restoration model might not have the capacity to learn the complex mapping from a corrupted image to that specific target perfectly.
  • Extreme Conditions: In cases of extremely heavy haze or near-total darkness, the restoration still struggles to recover enough information for the detector to function, leading to failure cases.
  • Future Direction: A promising next step would be exploring how to increase the "learnability" of the pseudo ground truth so that even lightweight restoration models can achieve these detection gains.

Summary updated

Abstract

Although image restoration has achieved significant progress, its potential to assist object detectors in adverse imaging conditions lacks enough attention. It is reported that the existing image restoration methods cannot improve the object detector performance and sometimes even reduce the detection performance. To address the issue, we propose a targeted adversarial attack in the restoration procedure to boost object detection performance after restoration. Specifically, we present an ADAM-like adversarial attack to generate pseudo ground truth for restoration training. Resultant restored images are close to original sharp images, and at the same time, lead to better results of object detection. We conduct extensive experiments in image dehazing and low light enhancement and show the superiority of our method over conventional training and other domain adaptation and multi-task methods. The proposed pipeline can be applied to all restoration methods and detectors in both one- and two-stage.
BibTeX
@inproceedings{sun2022rethinking,
  author = {Shangquan Sun and Wenqi Ren and Tao Wang and Xiaochun Cao},
  title = {{Rethinking Image Restoration for Object Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=se2oxj-6Nz},
}
Figure