The Tale of Evil Twins: Adversarial Inputs versus Backdoored Models

Ren Pang, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Ting Wang

Introduction

In the evolving landscape of AI security, researchers have traditionally treated adversarial inputs and poisoned models as distinct threats. Adversarial inputs involve subtle tweaks to data at inference time to fool a fixed model, while poisoning (or backdooring) involves modifying the model during training to respond to specific triggers. This paper challenges this siloed view, introducing a unified framework called Input Model Co-optimization (IMC). The authors demonstrate that these two attack vectors are "evil twins"—fundamentally linked and capable of being jointly optimized to create exploits far more potent than either used in isolation.

The IMC Framework

The IMC framework treats the attack as a bi-optimization problem. Instead of choosing between perturbing the data or the model, an adversary optimizes both simultaneously to balance three competing objectives:

  • Efficacy: The success rate in forcing a specific misclassification.
  • Fidelity: The perceptual quality of the adversarial input (ensuring it remains "clean" to human observers).
  • Specificity: The model's performance on normal, non-target inputs (avoiding noticeable accuracy drops on clean data).
The three competing goals of the IMC framework: Efficacy, Specificity, and Fidelity.

By alternating between updating the input perturbation and the model parameters, the IMC algorithm finds an optimal middle ground where minimal changes to both lead to maximum impact.

Mechanisms of Duality

The core insight of the paper is the geometric duality between input and model perturbations. In a standard classification task, an attacker can achieve a misclassification in two ways: by moving the input sample across the classification boundary, or by moving the boundary itself to encompass the input sample.

Comparison of Input Perturbation (moving the sample) and Model Perturbation (moving the classification boundary).

This duality leads to what the authors call the Leverage Effect. They found that sacrificing a tiny amount of one metric (e.g., allowing a slightly larger input perturbation) can lead to disproportionately large gains in another (e.g., significantly improving the model's specificity). On the GTSRB traffic sign dataset, a mere 5% increase in input noise allowed the attacker to reduce the model's accuracy drop on clean data by over 48%.

Key Results and Evasion

The power of co-optimization was evaluated across several benchmarks, including CIFAR10, Mini-ImageNet, and high-stakes medical (ISIC) and autonomous driving (GTSRB) datasets.

  • Amplified Efficacy: In the ISIC skin cancer screening dataset, while isolated attacks reached confidence scores of ~0.25 to ~0.4, the IMC attack achieved over 0.8 confidence under identical constraints.
  • Defeating Defenses: The framework proved resilient against gold-standard defenses like Adversarial Training. By leveraging the co-optimization process, attackers could regain success even when the model was specifically trained to resist noise.
  • Optimized Evasiveness: The authors developed TrojanNN, an optimized version of existing backdoor attacks. By tuning triggers to specific "abnormal" neurons, TrojanNN successfully evaded state-of-the-art detectors like ABS, dropping detection rates from nearly 100% to as low as 40%.

Conclusion and Future Directions

The study of IMC reveals that the current security paradigm of defending against inputs and models separately is fundamentally flawed. Sophisticated adversaries will likely exploit the synergy between these vectors. Future work must focus on developing unified defense metrics and ensemble approaches that can detect these co-optimized threats. As AI systems are deployed in increasingly critical roles, understanding the interaction between these "evil twins" is paramount for building truly robust and secure models.

Summary updated

BibTeX
@inproceedings{pang2020tale,
  author = {Ren Pang and Xinyang Zhang and Shouling Ji and Yevgeniy Vorobeychik and Xiapu Luo and Ting Wang},
  title = {{The Tale of Evil Twins: Adversarial Inputs versus Backdoored Models}},
  booktitle = {CCS},
  year = {2020},
  url = {https://openreview.net/forum?id=rngV3SQ8qV3},
}
Figure