Abstract
Neural networks are powering the deployment of embedded devices and Internet of Things. Applications range from personal assistants to critical ones such as self-driving cars. It has been shown recently that models obtained from neural nets can be trojaned; an attacker can then trigger an arbitrary model behavior facing crafted inputs. This has a critical impact on the security and reliability of those deployed devices. We introduce novel algorithms to detect the tampering with deployed models, classifiers in particular. In the remote interaction setup we consider, the proposed strategy is to identify markers of the model input space that are likely to change class if the model is attacked, allowing a user to detect a possible tampering. This setup makes our proposal compatible with a wide range of scenarios, such as embedded models, or models exposed through prediction APIs. We experiment those tampering detection algorithms on the canonical MNIST dataset, over three different types of neural nets, and facing five different attacks (trojaning, quantization, fine-tuning, compression and watermarking). We then validate over five large models (VGG16, VGG19, ResNet, MobileNet, DenseNet) with a state of the art dataset (VGGFace2), and report results demonstrating the possibility of an efficient detection of model tampering.
Introduction
As neural networks transition from research environments to embedded IoT devices and remote prediction APIs, ensuring their integrity becomes a critical security challenge. A deployed model might be maliciously "trojaned"—injected with a backdoor that triggers specific behaviors only when presented with a particular input pattern. Alternatively, service providers might silently apply compression or quantization to reduce costs, potentially altering the model's performance in ways not agreed upon by the user.
In "black-box" scenarios, where internal weights are inaccessible, standard accuracy metrics often fail to detect these stealthy modifications. TamperNN addresses this by monitoring the "behavioral heartbeat" of the model through a minimal set of sensitive queries called markers.
Methodology
The core strategy of TamperNN is to identify markers: specific inputs that are highly sensitive to model changes. While a model's overall accuracy might remain stable after tampering, its decision boundaries inevitably shift. The authors propose four strategies for marker selection:
- Strawman (SM): Randomly selected images from the original test set, serving as a baseline.
- GRID: Random noise inputs at the corners of the input hypercube. This model-agnostic approach is effective because models are less stable in these out-of-distribution regions.
- Weight Perturbation (WGHT): A "white-box" preparation method where markers are chosen based on their sensitivity to random noise added to the model's weights.
- Boundary Adversarial (BADV): Using adversarial techniques to find inputs sitting precisely on the decision boundary. These "edge" points are the most likely to flip classes even with minor weight shifts.
Experimental Evaluation
The methods were evaluated using the MNIST dataset across various architectures (MLP, CNN, IRNN) against five types of modifications: quantization, compression, fine-tuning, watermarking, and trojaning. Further validation was performed on the VGGFace2 dataset using large-scale models including VGG16, VGG19, ResNet50, MobileNet, and DenseNet121.
Key Results
- High Sensitivity: The proposed algorithms (
BADV,WGHT, andGRID) significantly outperformed random sampling. For instance,WGHTdetected compression attacks with a marker triggering ratio 100x higher than the baseline. - Efficiency: For a trojaned VGG-Face model, while overall accuracy dropped by only 2.35%,
GRIDmarkers achieved a detection ratio of ~45%. This implies that a very small set of queries (e.g., s=10) can provide high statistical confidence in detecting tampering. - Robustness: The approach proved effective across diverse architectures, demonstrating its general applicability to modern deep learning deployments.
Discussion and Conclusion
TamperNN demonstrates that decision boundaries are fragile indicators of model integrity. By focusing on out-of-distribution noise or boundary-adjacent points, users can detect subtle tampering that standard benchmarks miss. While the method requires the challenger to have had access to the original model to establish a baseline, it offers a powerful, low-overhead solution for remote integrity verification. Future work could explore adaptive attackers who might attempt to identify and bypass marker queries.
Summary updated