Trojan Model Detection Using Activation Optimization

Mohamed E. Hussein, Sudharshan Subramaniam Janakiraman, Wael AbdAlmageedarXiv 2023 · arXiv · OpenReview

Abstract

Due to data's unavailability or large size, and the high computational and human labor costs of training machine learning models, it is a common practice to rely on open source pre-trained models whenever possible. However, this practice is worrisome from the security perspective. Pre-trained models can be infected with Trojan attacks, in which the attacker embeds a trigger in the model such that the model's behavior can be controlled by the attacker when the trigger is present in the input. In this paper, we present our preliminary work on a novel method for Trojan model detection. Our method creates a signature for a model based on activation optimization. A classifier is then trained to detect a Trojan model given its signature. Our method achieves state of the art performance on two public datasets.

Introduction

As training high-performance deep learning models becomes prohibitively expensive, the industry has shifted toward using open-source, pre-trained models. This creates a massive security loophole: Trojan (or backdoor) attacks. In a Trojan attack, an adversary poisons a small fraction of the training data so the model learns to associate a specific "trigger" (like a small patch or a specific pattern) with a "target class." The model behaves perfectly on normal data but flips to the malicious target class whenever the trigger is present.

TRIGS (TRojan Identification from Gradient-based Signatures) is a novel, data-driven framework designed to unmask these backdoored models. Instead of attempting to reverse-engineer the trigger—a process that is often computationally expensive and architecture-specific—TRIGS generates a unique "signature" for the model through activation optimization. By analyzing how a model reacts when forced to minimize or maximize specific class outputs, TRIGS can identify the subtle fingerprints left by an attacker.

Methods: Generating Model Signatures

The core of the TRIGS approach is the Model Signature. Rather than inspecting model weights directly, the framework observes how the model "dreams" or visualizes features through gradient-based optimization.

Figure 1: Overview of the TRIGS detection pipeline, from activation optimization to final classification.

The process follows four main steps:

  1. Activation Optimization: For a $K$-class model, $M$ loss functions are optimized to synthesize images that either maximize or minimize the logits for each class.
  2. Signature Generation: These synthesized images constitute the model's signature. If a model is Trojaned, the optimization process tends to "leak" the trigger pattern into these images, even if the defender is not specifically looking for it.
  3. Feature Extraction: To handle high-dimensional models (e.g., ImageNet with 1,000 classes), pixel-wise statistics (mean, standard deviation, quantiles, and histograms) are calculated across the signature images. This reduces the data into a fixed-size feature map.
  4. Detection: A binary classifier (specifically a ResNeXt-50) is trained on these signatures or statistical features to distinguish between "Benign" and "Trojan" models.

Datasets and Experimental Setup

The authors evaluated TRIGS on three primary benchmarks:

  • CIFAR10: Models based on a modified VGG architecture.
  • Tiny ImageNet: Models based on a shallow ResNet10 architecture.
  • ImageNet-ViT (New): A challenging new dataset consisting of 1,200 Vision Transformer (ViT-B-16) models trained on the full ImageNet (1000 classes). This is the first large-scale dataset for Trojan detection focusing on the ViT architecture.

Results and Key Insights

TRIGS achieved state-of-the-art (SOTA) performance across all tested benchmarks. On CIFAR10 and Tiny ImageNet, the "Stats" variant of TRIGS achieved an AUC between 0.99 and 1.00, significantly outperforming existing baselines.

Figure 6: Performance comparison (AUC) of TRIGS variants against baselines on the CIFAR10 dataset.

Several key insights emerged from the study:

  • Minimization is Effective: Activation minimization (minimizing class output) was often more effective at revealing Trojan fingerprints than maximization, a departure from traditional model inversion techniques.
  • Architecture Agnostic: Because TRIGS operates on input/output relationships via gradients, it works equally well on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). On the ImageNet-ViT dataset, TRIGS achieved an AUC of 0.94, while the best baseline (MM-BD) struggled at 0.59.
  • Efficiency: The use of pixel-wise statistics allows the method to remain scalable even for models with a large number of classes.

Limitations and Future Work

While effective, TRIGS has some limitations. Generating the initial signature requires running gradient-based optimization for every class, which can be computationally heavy for very large models. Additionally, training the detector requires "shadow models" (benign and Trojaned versions of the task model), assuming the defender has the resources to train them. Future research will explore "detection-aware" Trojan attacks designed to hide fingerprints during optimization.

Summary updated

Abstract

Due to data's unavailability or large size, and the high computational and human labor costs of training machine learning models, it is a common practice to rely on open source pre-trained models whenever possible. However, this practice is worry some from the security perspective. Pre-trained models can be infected with Trojan attacks, in which the attacker embeds a trigger in the model such that the model's behavior can be controlled by the attacker when the trigger is present in the input. In this paper, we present our preliminary work on a novel method for Trojan model detection. Our method creates a signature for a model based on activation optimization. A classifier is then trained to detect a Trojan model given its signature. Our method achieves state of the art performance on two public datasets.
BibTeX
@misc{hussein2023trojan,
  author = {Mohamed E. Hussein and Sudharshan Subramaniam Janakiraman and Wael AbdAlmageed},
  title = {{Trojan Model Detection Using Activation Optimization}},
  year = {2023},
  eprint = {2306.04877},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.04877},
}
Figure