TamperNN: Efficient Tampering Detection of Deployed Neural Nets

Erwan Le Merrer, Gilles Trédan

TL;DR

TamperNN introduces a "black-box" method to detect if a deployed neural network has been modified—whether by malicious trojaning or standard operations like quantization—without needing access to the model's internal weights. By identifying and querying a small set of highly sensitive "marker" inputs, users can detect subtle changes in decision boundaries that indicate the model is no longer the original version.

What Problem Does This Paper Address?

As neural networks move from research labs to embedded IoT devices and remote prediction APIs, a critical security gap emerges: integrity. How do you know the model running on a remote server or a smart camera is exactly the one you trained?

An attacker might "trojan" a model, injecting a back door that triggers a specific behavior only when a certain pattern is present in the input. Alternatively, a service provider might silently compress or quantize a model to save costs, potentially degrading performance in ways the user didn't agree to.

In a "black-box" scenario, you cannot inspect the model's weights or code; you can only send inputs and receive classification labels. Standard accuracy metrics are often too blunt to catch these changes, as many attacks (like trojaning) are designed to maintain high accuracy on normal data. The authors tackle the challenge of detecting these "stealthy" modifications using a minimal number of queries.

Methods

The authors propose moving away from checking general accuracy and instead focusing on markers: specific inputs that are likely to change their predicted class if the model is even slightly tampered with. They compare four strategies for selecting these markers:

  1. SM (Strawman): Randomly selecting images from the original test set. This is the baseline.
  2. GRID: Generating random noise inputs (specifically, random bits at the corners of the input hypercube). This is model-agnostic and surprisingly effective because these points are far from the training data distribution where decision boundaries are most stable.
  3. WGHT (Weight Perturbation): A "white-box" preparation for a "black-box" deployment. The model owner identifies inputs that change labels when the model's weights are slightly jittered with random noise.
  4. BADV (Boundary Adversarial): Using adversarial attack techniques (like FGSM) to find inputs that sit exactly on the decision boundary. Because they are on the "edge," even a tiny shift in the model's weights will likely push these points into a different class.

Datasets

The researchers evaluated their methods using:

  • MNIST: The classic handwritten digit dataset, used to test three different architectures (MLP, CNN, and IRNN) against five types of "attacks" (quantization, flooring/compression, fine-tuning, watermarking, and trojaning).
  • VGGFace2: A large-scale face recognition dataset used to validate the approach on five state-of-the-art large models: VGG16, VGG19, ResNet50, MobileNet, and DenseNet121.

Key Insights and Ideas

  • Decision Boundaries are Fragile: The core "aha" moment is that while a model's overall accuracy might be robust, its specific decision boundaries are highly sensitive. Tampering—even for benign reasons like compression—inevitably shifts these boundaries.
  • The "Marker" Concept: Instead of a 10,000-image test suite, you can use a "key" of just 10 to 100 markers. If the model's response to these markers changes, the "heartbeat" has skipped a beat, and the model is compromised.
  • Out-of-Distribution Effectiveness: The GRID method's success suggests that models are much less stable when processing random noise than when processing structured data like photos. This makes noise a great candidate for "integrity markers."
  • Efficiency vs. Stealth: There is a fundamental tradeoff between an attacker’s desire to remain stealthy (minimal weight changes) and the challenger's ability to detect them. The authors show that by picking points on the boundary (BADV), you can detect attacks that are two orders of magnitude smaller than what random sampling can catch.

Key Results

  • Superior Detection: The proposed algorithms (BADV, WGHT, and GRID) significantly outperformed the strawman approach across all models and attacks.
  • Sensitivity: In one experiment, the WGHT algorithm detected a compression attack with a marker triggering ratio 100x higher than the strawman.
  • Low Overhead: For a trojaned VGG-Face model, while the attack only degraded overall accuracy by 2.35%, the GRID markers achieved a detection ratio of ~45%, meaning a very small set of queries (s=10) would be virtually guaranteed to catch the tampering.
  • Scalability: The methods held up across diverse architectures, from simple MLPs to complex ResNets, proving the approach is generally applicable to modern deep learning.

Limitations and Future Work

  • The "Original" Requirement: This method requires the "challenger" (the person checking the model) to have had access to the original, untampered model at some point to generate the markers and record the baseline responses.
  • Detection Floor: The authors admit that if an attack is extremely subtle (e.g., an accuracy change of only 0.02%), even their best algorithms might fail to find a marker that flips.
  • Adaptive Attackers: The paper doesn't deeply explore "aware" attackers who might try to detect if they are being queried with markers and return the "correct" (original) label to avoid detection.
  • Future Directions: The authors suggest exploring even more sensitive marker generation techniques and investigating how these methods might be integrated into "remote attestation" protocols for secure hardware.

Summary updated

Abstract

Neural networks are powering the deployment of embedded devices and Internet of Things. Applications range from personal assistants to critical ones such as self-driving cars. It has been shown recently that models obtained from neural nets can be trojaned; an attacker can then trigger an arbitrary model behavior facing crafted inputs. This has a critical impact on the security and reliability of those deployed devices. We introduce novel algorithms to detect the tampering with deployed models, classifiers in particular. In the remote interaction setup we consider, the proposed strategy is to identify markers of the model input space that are likely to change class if the model is attacked, allowing a user to detect a possible tampering. This setup makes our proposal compatible with a wide range of scenarios, such as embedded models, or models exposed through prediction APIs. We experiment those tampering detection algorithms on the canonical MNIST dataset, over three different types of neural nets, and facing five different attacks (trojaning, quantization, fine-tuning, compression and watermarking). We then validate over five large models (VGG16, VGG19, ResNet, MobileNet, DenseNet) with a state of the art dataset (VGGFace2), and report results demonstrating the possibility of an efficient detection of model tampering.
BibTeX
@inproceedings{merrer2019tampernn,
  author = {Erwan Le Merrer and Gilles Trédan},
  title = {{TamperNN: Efficient Tampering Detection of Deployed Neural Nets}},
  booktitle = {ISSRE},
  year = {2019},
  url = {https://openreview.net/forum?id=Y9A6CgHcNY},
}

Loading related content...

Figure