Hindering Adversarial Attacks with Implicit Neural Representations

Andrei A. Rusu, Dan A. Calian, Sven Gowal, Raia Hadsell

TL;DR

This paper introduces LINAC, a novel defense mechanism that uses Implicit Neural Representations (INRs) to transform images before feeding them into a classifier. This transformation, controlled by a secret "private key," significantly hinders common adversarial attacks without requiring adversarial training of the classifier itself. The authors also propose a new attack strategy, PBA, to challenge these key-based defenses.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial attacks, where imperceptible changes to input data can cause misclassifications. While many defenses exist, they often fall into two categories: making decision boundaries insensitive to perturbations (requiring adversarial training) or making it computationally expensive to find adversarial examples. The latter approach, while theoretically sound, has seen many defenses circumvented by more sophisticated attacks. This paper aims to develop a defense that leverages computational hardness through input transformation, specifically by using INRs, to hinder adversarial attacks without the need for extensive adversarial training.

Methods

The core of the proposed defense is the Lossy Implicit Network Activation Coding (LINAC). This method takes an RGB image and fits an Implicit Neural Representation (INR) – essentially a small neural network – to approximate the pixel color intensities. Instead of using the INR's output directly, LINAC uses the activations from an intermediate hidden layer of the INR as the transformed input for a standard classifier. This transformation is deterministic but "lossy," meaning it doesn't perfectly reconstruct the original image. Crucially, the initialization of the INR and the shuffling of pixels during training are controlled by a private key (a random seed). This key is essential for generating the correct transformation; without it, attacks are significantly hampered.

The authors also introduce a novel attack strategy called Parametric Bypass Approximation (PBA). This attack aims to circumvent defenses that rely on input transformations by learning a parametric approximation of the unknown transformation. PBA uses the defended classifier's parameters, training data, and loss function to train a surrogate model that mimics the transformation, allowing for more effective adversarial example generation.

Datasets

The experiments were conducted on the CIFAR-10 dataset.

Key Insights and Ideas

  • Private Key as a Defense Mechanism: The random seed used to initialize the INR and control data shuffling acts as a private key. This key is crucial for generating the correct LINAC transformation. Without it, adversarial attacks are significantly weakened, suggesting a novel way to leverage computational hardness for defense.
  • Lossy INR for Robustness: Using intermediate, "lossy" activations from an INR, rather than a perfect reconstruction, appears to be key to hindering adversarial attacks. This suggests that the information preserved by the INR is sufficient for classification but not for generating effective adversarial perturbations.
  • PBA Attack's Effectiveness: The PBA attack demonstrates that even sophisticated input transformations can be vulnerable if the attacker has access to the classifier's parameters and training data. This highlights the importance of considering adaptive attacks that can learn to bypass defenses.
  • LINAC's Agnosticism to Attack Norms: LINAC shows a notable difference in its effectiveness against L∞ and L2 norm perturbations, hindering L2 attacks more significantly. This suggests that the nature of the perturbation matters and that LINAC's defense is not uniform across all attack types.

Key Results

  • LINAC successfully hinders common adversarial attacks on CIFAR-10 for perturbations up to $\epsilon = 8/255$ in L∞ norm and $\epsilon = 0.5$ in L2 norm.
  • Classifiers trained on LINAC-transformed data achieve high clean accuracy (around 93%) without adversarial training.
  • Attacks without access to the private key (e.g., using random keys) result in significantly reduced classifier accuracy (around 30% on average).
  • The novel PBA attack strategy successfully invalidates existing key-based defenses and is more effective than other adaptive attacks against LINAC.
  • LINAC's defense is more effective against L2 norm attacks than L∞ norm attacks.

Limitations and Future Work

  • Computational Cost: The fitting of INRs for LINAC is computationally expensive, comparable to inference with a WideResNet-70-16 model. This cost is dominated by the INR fitting process.
  • Lossy Representation: LINAC leads to lossy image approximations, meaning some information is lost. While this aids defense, it might impact performance on complex tasks or very high-resolution images.
  • Key Dependency: The defense relies heavily on the secrecy of the private key. If the key is compromised, the defense is significantly weakened.
  • Future Work: The authors suggest exploring adaptive "early stopping" for INR fitting to reduce computational overhead, investigating INR research for efficiency gains, and scaling LINAC to larger images. They also plan to explore the impact of different INR architectures and representation layer choices.

Summary updated

Abstract

We introduce the Lossy Implicit Network Activation Coding (LINAC) defence, an input transformation which successfully hinders several common adversarial attacks on CIFAR-$10$ classifiers for perturbations up to $\epsilon = 8/255$ in $L_\infty$ norm and $\epsilon = 0.5$ in $L_2$ norm. Implicit neural representations are used to approximately encode pixel colour intensities in $2\text{D}$ images such that classifiers trained on transformed data appear to have robustness to small perturbations without adversarial training or large drops in performance. The seed of the random number generator used to initialise and train the implicit neural representation turns out to be necessary information for stronger generic attacks, suggesting its role as a private key. We devise a Parametric Bypass Approximation (PBA) attack strategy for key-based defences, which successfully invalidates an existing method in this category. Interestingly, our LINAC defence also hinders some transfer and adaptive attacks, including our novel PBA strategy. Our results emphasise the importance of a broad range of customised attacks despite apparent robustness according to standard evaluations. LINAC source code and parameters of defended classifier evaluated throughout this submission are available: https://github.com/deepmind/linac
BibTeX
@inproceedings{rusu2022hindering,
  author = {Andrei A. Rusu and Dan A. Calian and Sven Gowal and Raia Hadsell},
  title = {{Hindering Adversarial Attacks with Implicit Neural Representations}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=wpSZ5lTsIbf},
}
Figure