The Taboo Trap: Behavioural Detection of Adversarial Samples

Ilia Shumailov, Yiren Zhao, Robert D. Mullins, Ross AndersonarXiv 2018 · OpenReview

Abstract

Deep Neural Networks (DNNs) have become a powerful tool for a wide range of problems. Yet recent work has found an increasing variety of adversarial samples that can fool them. Most existing detection mechanisms against adversarial attacks impose significant costs, either by using additional classifiers to spot adversarial samples, or by requiring the DNN to be restructured. In this paper, we introduce a novel defence. We train our DNN so that, as long as it is working as intended on the kind of inputs we expect, its behavior is constrained, in that some set of behaviors are taboo. If it is exposed to adversarial samples, they will often cause a taboo behavior, which we can detect. Taboos can be both subtle and diverse, so their choice can encode and hide information. It is a well-established design principle that the security of a system should not depend on the obscurity of its design, but on some variable (the key) which can differ between implementations and be changed as necessary. We discuss how taboos can be used to equip a classifier with just such a key, and how to tune the keying mechanism to adversaries of various capabilities. We evaluate the performance of a prototype against a wide range of attacks and show how our simple defense can defend against cheap attacks at scale with zero run-time computation overhead, making it a suitable defense method for IoT devices.

Introduction

Deep Neural Networks (DNNs) are notoriously fragile. Small, human-imperceptible perturbations to an input—known as adversarial samples—can cause a model to confidently misclassify an image or command. While many defenses exist, they usually come with a heavy "tax" in terms of computational cost (requiring auxiliary detector networks) or structural complexity (needing fundamental architecture redesigns).

These overheads are often deal-breakers for Internet of Things (IoT) and edge devices, where battery life and processing power are strictly limited. The "Taboo Trap" is a lightweight defense mechanism designed to provide "situational awareness"—knowing when the system is under attack—without slowing down the model or increasing its memory footprint.

The Taboo Trap Method

The Taboo Trap treats internal network activations as a security feature. The core intuition is that adversarial perturbations often push internal activations into extreme ranges that clean data never touches. By "trapping" these extremes, the system can spot an attacker. The process follows three main steps:

  1. Profiling: Clean data is passed through a pre-trained model to observe the "normal" range of activation values across different layers.
  2. Constraint Training: A "transform function" (the Taboo) specifies forbidden activation ranges. For example, a rule might state that no neuron in a specific layer should exceed its 99th percentile value. The model is then fine-tuned using a regularizer that penalizes activations falling into these "taboo" zones.
  3. Detection: At runtime, if any activation value exceeds the pre-set threshold, the system triggers an alarm. This requires only a simple value comparison (e.g., if activation > threshold: alarm()), adding effectively zero latency.

Crucially, the choice of monitored layers and thresholds acts as a cryptographic key. Following Kerckhoffs’s Principle, the system remains secure even if the attacker knows the defense mechanism, as long as they do not know the specific "key" (the taboo thresholds) chosen by the defender.

Experimental Evaluation

The researchers evaluated the Taboo Trap using standard computer vision benchmarks, including MNIST (LeNet5) and CIFAR10 (M-CifarNet and ResNet18). The models were tested against three adversarial attacks: FGSM (a fast, single-step attack), PGD (a stronger iterative attack), and DeepFool (an optimized attack for minimal perturbations).

Results and Analysis

The Taboo Trap demonstrates high effectiveness against "cheap" attacks like FGSM, which are the most likely to be deployed at scale. On the MNIST dataset, detection rates reached as high as 99%.

Detection performance (A: Accuracy, D: Detection, AD: Combined) across different models and attack parameters.

A standout feature of this approach is its efficiency. Unlike competing defenses like MagNet or SafetyNet, which introduce significant computational overhead, the Taboo Trap adds zero extra Multiply-Accumulate operations (MACs) and zero extra parameters.

Comparison of Taboo Trap against other defense mechanisms, highlighting its zero-overhead profile.

Key insights from the results include:

  • Performance Preservation: Training with taboo constraints did not degrade the model's accuracy on clean data.
  • Model Capacity: Larger models like ResNet18 showed better defense performance, suggesting they have more "room" to learn taboo behaviors without sacrificing primary task performance.

Conclusion and Future Work

While highly effective against single-step attacks, the Taboo Trap is less robust against sophisticated, iterative attacks like DeepFool, which can sometimes navigate around the "forbidden" activation ranges. The authors suggest that future work could involve more complex "keys"—such as non-linear taboo ranges or combining multiple transform functions—to further harden the trap against white-box attackers. Ultimately, the Taboo Trap serves as an efficient first line of defense, particularly suited for resource-constrained environments.

Summary updated

Abstract

Deep Neural Networks (DNNs) have become a powerful toolfor a wide range of problems. Yet recent work has found an increasing variety of adversarial samplesthat can fool them. Most existing detection mechanisms against adversarial attacksimpose significant costs, either by using additional classifiers to spot adversarial samples, or by requiring the DNN to be restructured. In this paper, we introduce a novel defence. We train our DNN so that, as long as it is workingas intended on the kind of inputs we expect, its behavior is constrained, in that some set of behaviors are taboo. If it is exposed to adversarial samples, they will often cause a taboo behavior, which we can detect. Taboos can be both subtle and diverse, so their choice can encode and hide information. It is a well-established design principle that the security of a system should not depend on the obscurity of its design, but on some variable (the key) which can differ between implementations and bechanged as necessary. We discuss how taboos can be used to equip a classifier with just such a key, and how to tune the keying mechanism to adversaries of various capabilities. We evaluate the performance of a prototype against a wide range of attacks and show how our simple defense can defend against cheap attacks at scale with zero run-time computation overhead, making it a suitable defense method for IoT devices.
BibTeX
@misc{shumailov2018taboo,
  author = {Ilia Shumailov and Yiren Zhao and Robert D. Mullins and Ross Anderson},
  title = {{The Taboo Trap: Behavioural Detection of Adversarial Samples}},
  year = {2018},
  eprint = {1811.07375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.07375},
}
Figure