Towards Adversarial Patch Analysis and Certified Defense against Crowd Counting

Qiming Wu, Zhikang Zou, Pan Zhou, Xiaoqing Ye, Binghui Wang, Ang Li

TL;DR

Researchers have developed a potent new way to fool crowd-counting AI using small, printable "adversarial patches" that can cause systems to wildly over- or under-count people. To counter this, they introduced Randomized Ablation (RA), the first certified defense for regression-based crowd counting that provides a mathematical guarantee of robustness by training models on heavily "sampled" versions of images.

What Problem Does This Paper Address?

Crowd counting is a high-stakes application of Deep Neural Networks (DNNs), used for everything from monitoring social distancing to preventing deadly stampedes in stadiums. While we know DNNs are vulnerable to "adversarial examples" (tiny, invisible pixel tweaks), these are hard to deploy in the real world.

An adversarial patch—a visible, physical sticker placed in a scene—is a much more practical threat. However, most research on patches focuses on classification (changing a "dog" to a "cat"). This paper tackles the more complex world of regression, where the goal is to disrupt a continuous numerical output (the person count). The authors highlight a dangerous gap: current crowd-counting models are highly susceptible to these patches, and traditional defenses like Adversarial Training (ADT) fail to provide reliable protection.

Methods

The authors propose a two-pronged approach: a new attack to test limits and a new defense to secure them.

1. APAM (Adversarial Patch Attack with Momentum)

To create a "universal" patch that works regardless of where it's placed or how the camera is angled, the authors developed APAM.

  • Image Transformations: During the patch's creation, it is digitally rotated, scaled, and interpolated. This ensures the patch remains effective even when printed and viewed through a real surveillance camera.
  • Momentum-based Optimization: They use momentum in the gradient descent process to avoid getting stuck in "local minima," resulting in a more stable and powerful adversarial perturbation.
  • Black-box Attack: They demonstrate that a patch generated using one set of models can successfully "transfer" to fool a completely different, unknown model.

2. Randomized Ablation (RA)

This is the paper's core defensive contribution. Instead of training on full images, the model is trained on images where only a small number of pixels ($k$) are retained, and the rest are set to zero (ablated).

  • The Logic: If an adversarial patch only covers a small percentage of an image, there is a high statistical probability that the "randomized ablation" process will remove most or all of the patch's pixels, while still leaving enough of the original scene for the model to count the crowd.
  • Certification: Unlike empirical defenses that just "seem to work," RA provides a theoretical lower bound on how much an attacker can possibly disrupt the count.

Datasets

The study primarily utilizes the ShanghaiTech dataset (Part A), which is a gold standard in the field containing 1,198 images with over 330,000 annotated people in congested scenes. They also tested the attack's "transferability" across five major model architectures: CSRNet, DA-Net, MCNN, CAN, and CMTL.

Key Insights and Ideas

  • Background Exploitation: Unlike previous attacks that focused on depth information, APAM leverages the "extremely dense background information" of congested scenes to hide its influence.
  • The Fragility of Regression: A patch covering less than 6% of the image can cause the count to swing by hundreds. Interestingly, different models fail in different ways: some (like CSRNet) tend to over-count massively, while others (like MCNN) are tricked into seeing almost zero people.
  • Defense Trade-offs: There is a "no free lunch" theorem at play. While Randomized Ablation makes a model incredibly robust to attacks, it slightly reduces accuracy on "clean" (non-attacked) images. However, the authors argue this is a necessary trade-off for safety-critical systems.

Key Results

  • Attack Potency: In white-box settings, patches as small as 163x163 pixels increased the Mean Absolute Error (MAE) of the CSRNet model from 68.20 to a staggering 463.53.
  • Defense Superiority: Randomized Ablation (RA) significantly outperformed standard Adversarial Training (ADT). On adversarial examples, the MAE for RA was 30 points lower than ADT, proving it is far more effective at neutralizing patches.
  • Physical World Success: The authors printed a patch and placed it in a real shopping mall. Even in this physical environment, the models' predictions were "strongly perturbed," with some models showing an error rate of over 900%.

Limitations and Future Work

The authors acknowledge that their physical attacks, while successful, are "somewhat weak" compared to digital ones. Real-world variables like lighting conditions, extreme camera angles, and human interference make designing a truly "universal" physical patch difficult.

Future work likely involves refining the Randomized Ablation technique to close the "accuracy gap" on clean images and exploring how to make patches even more robust to the chaotic lighting and environmental factors of real-world surveillance.

Summary updated

Abstract

Crowd counting has drawn much attention due to its importance in safety-critical surveillance systems. Especially, deep neural network (DNN) methods have significantly reduced estimation errors for crowd counting missions. Recent studies have demonstrated that DNNs are vulnerable to adversarial attacks, i.e., normal images with human-imperceptible perturbations could mislead DNNs to make false predictions. In this work, we propose a robust attack strategy called Adversarial Patch Attack with Momentum (APAM) to systematically evaluate the robustness of crowd counting models, where the attacker's goal is to create an adversarial perturbation that severely degrades their performances, thus leading to public safety accidents (e.g., stampede accidents). Especially, the proposed attack leverages the extreme-density background information of input images to generate robust adversarial patches via a series of transformations (e.g., interpolation, rotation, etc.). We observe that by perturbing less than 6% of image pixels, our attacks severely degrade the performance of crowd counting systems, both digitally and physically. To better enhance the adversarial robustness of crowd counting models, we propose the first regression model-based Randomized Ablation (RA), which is more sufficient than Adversarial Training (ADT) (Mean Absolute Error of RA is 5 lower than ADT on clean samples and 30 lower than ADT on adversarial examples). Extensive experiments on five crowd counting models demonstrate the effectiveness and generality of the proposed method.
BibTeX
@inproceedings{wu2021towards,
  author = {Qiming Wu and Zhikang Zou and Pan Zhou and Xiaoqing Ye and Binghui Wang and Ang Li},
  title = {{Towards Adversarial Patch Analysis and Certified Defense against Crowd Counting}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=toNXFmc5Rb},
}

Loading related content...

Figure